REMARKS 

Claims 1-36 are pending in the present application. Claims 1-36 stand rejected. 
Claims 1-21, 23, 27, 31, 33, 34 and 36 are herein amended. 

In accordance with 37 C.F.R. §1.1 21(c), a clean version of the specification and 
of the pending claims is provided hereinabove and a marked up version of the 
specification and the claims being changed by the current amendment is attached hereto. 

The Examiner objected to the drawings under 37 C.F.R. § 1.83(a), stating that 
Figures 8-14 were addressed in the specification but not included with the application. 
Applicants submit herewith a complete set of drawings of Figures 1-14. Applicants 
submit that no new matter has been added. Accordingly, the objection to the drawings is 
believed to have been overcome. 

The Examiner objected to the title as being non-descriptive. Applicants have 
amended the title to read "Method and System for Testing Software Components". The 
Examiner's objection to the title is believed to have been overcome. 

The Examiner objected to the specification for a variety of reasons, requesting a 
new specification. The specification has been amended as requested by the Examiner. 

The Examiner rejected claims 1-12, 14, 19 and 36 under 35 U.S.C. §112, second 
paragraph, as being indefinite. Specifically, the Examiner objected to the use of a 
trademark in the claims. Claims 1-12, 14, 19 and 36 have been amended to include the 
term "software implementation of a processor" in place of the trademark "JVM" 
previously used. Accordingly, the rejection of claim claims 1-12, 14, 19 and 36 is 
believed to have been overcome. 

The Examiner has made a provisional double patenting rejection regarding 
claims 1, 3, 13, 17, 18, 23 and 31 over claims of co-pending application no. 09/482178. 
Upon an indication of allowance, Applicants will promptly file a terminal disclaimer. 
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The Examiner rejected claims 1-23, 27, 31 and 34 under 35 U.S.C. § 102(e) as 
being anticipated by U.S. Patent No. 6,466,120 to Dantressangle (hereinafter 
Dantressangle). Dantressangle uses a client computer to perform as one or more virtual 
user. A test script is used with one or more "virtual users" to stress the server by 
accessing data stored on a storage device connected to the server. 

In contrast to Dantressangle, the present invention tests technology based 
software components. An application (such as a commercial web site like amazon.com) 
comprises several technology based software components (such as ENTERPRISE JAVA 
BEAN technology components) which are cobbled together to provide the application. 
Dantressangle tests a server, not the individual technology based software components 
that make up the application on the server. The present invention tests the individual 
technology based software components that make up an application. In this manner 
much more detail can be obtained regarding the performance of the component. While 
Dantressangle can determine that the performance of the server degrades after 100 users 
are applied, for example, Dantressangle cannot tell you which software component of the 
application is the root cause for the performance degradation. The present invention, by 
testing the software components of the application, can identify the software component 
that is causing the degradation in performance, such that the performance of the 
application can be enhanced (e.g., by replacing the software component with a more 
robust component that provides the same function, or by use of multiple instances of the 
software component to provide a type of load balancing). 

Independent claims 1, 7, 13, 18, 23, 27, 31 and 34 have been amended to recite 
that a software component of an application under test is being tested, therefore claims 1, 
7, 13, 18, 23, 27, 31 and 34 are believed allowable over Dantressangle. Claims 2-6, 8-12, 
14-17, 19-22 depend from claims 1, 7, 13 and 18 and are believed allowable as they 
depend from a base claim which is allowable. Accordingly, the rejection of claims 1- 
23, 27, 31 and 34 under § 102(e) as being anticipated by Dantressangle is believed to have 
been overcome. 
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The Examiner rejected claims 24-26 and 28-30 under 35 U.S.C. § 103(a) as being 
unpatentable over Dantresssangle in view of U.S. Patent No. 6,401,220 to Grey et al. 
(hereinafter Grey). Claims 24-26 and 28-30 depend from claims 23 and 27 and are 
believed allowable as they depend fro a base claim which is believed allowable. 
Accordingly, the rejection of claims 24-26 and 28-30 under § 103(a) as being inpatentable 
over Dantressangle in view of Grey is believed to have been overcome. 

The Examiner rejected claims 32-33 and 35-36 under 35 U.S.C. § 103(a) as being 
unpatentable over Dantresssangle in view of U.S. Patent No. 6,237,135 to Timbol 
(hereinafter Timbol). Claims 32-33 and 35-36 depend from claims 31 and 34 and are 
believed allowable as they depend from a base claim which is believed allowable. 
Accordingly, the rejection of claims 32-33 and 35-36 under § 103(a) as being 
unpatentable over Dantressangle in view of Timbol is believed to have been overcome. 

The prior art made of record is not believed to disclose or suggest the present 
invention. 

In view of the above, the Examiners objections and rejections are believed to have 
been overcome, placing claims 1-36 in condition for allowance, and reconsideration and 
allowance thereof is respectfully requested. The Examiner is respectfully invited to 
telephone the undersigning attorney if there any questions regarding this Amendment or 
this application. 
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The Assistant Commissioner is hereby authorized to charge payment of any 
additional fees associated with this communication or credit any overpayment to Deposit 
Account No. 50-0845. 

Respectfully submitted, 

Daly, Crowley & Mofford, LLP 


Dated: J^>^/^ 2, 


275 Turnpike Street - Suite 101 
Canton, MA 02021-2310 
Telephone: (781) 401-9988 x24 
Facsimile: (781)401-9966 


Bv: /^^ Z^^^^>n 

Kermit Robinson 
Reg. No. 48,734 
Attorney for Applicant(s) 
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Specification with changes shown: 

METHOD AND SYSTEM FOR SOFTWARE OBJECT TESTING 

This inv e ntion r e lat e s g e n e rally to comput e r softwar e applications and mor e 

sp e cifically to testing comput e r software applications, 

Distribut e d computing has b e en used for many y e ars. Distribut e d computing is 

very prevalently used in " e nt e rprise - wid e " applications. An e nt e rprise wide appl ication 
is an application that allows a larg e group of p e opl e to work togeth e r on a common task. 
U s ually, an ent e rpris e wid e application performs functions that are e ssential to a 
company's busin e ss. For e xample, in a bank, p e opl e at e v e ry bank branch must b e abl e 
to access a databas e of accounts for e very bank customer. Likewise, at an insurance 
company, p e opl e all ov e r th e company must be abl e to access a databas e containing 
information about every policyholder. The softwar e that performs th e s e functions is 
generally known as enterpris e wid e applications. 

As availabl e hardware and software has e volved, th e architecture of enterpris e 
wid e applications has changed. An architectur e which is curr e ntly popular is call e d th e 
N Ti e r e nterpris e model. The most prevalent N tier enterpris e model is a three ti e r 
mod e l. The thr ee tiers ar e the front e nd, th e middl e ware and th e back end. Th e back e nd 
is the database. Th e front end is sometim e s referred to as a "cli e nt" or a Graphical Us e r 
Int e rface (GUI). The middlewar e is the software that manag e s interactions with the 
databas e and captur e s the "business logic." Bu s iness logic t e lls the syst e m how to 
validat e , proc e ss and report on the data in a fashion that is us e ful for th e p e opl e in th e 
e nt e rpris e . 

Th e middl e war e r e sid e s on a comput e r call e d a server. The database might be on 
the same computer or a different computer. The "client" is usually on an individual's 
p e rsonal comput e r. All of th e comput e rs are conn e ct e d tog e th e r through a n e twork. 
B e caus e many people use the ent e rpris e wide application, s uch systems are set up to 
allow simultan e ous users and ther e would b e many clients conn e ct e d to a single serv e r. 
Often, many clients will be connect e d to th e serv e r simultan e ously. 

Thos e familiar with Intern e t comm e rc e wall r e cogniz e that th e N ti e r e d mod el 
also describes many Int e rnet web sites that sell goods or s e rvices. For example, a web 
sit e that auctions cars is likely to fit th e N ti e r e d mod e l. In such an application, databas e s 
ar e provided to track buyers, sellers and objects being auctioned. Also, a database must 
b e provid e d to track the bids as th e y ar e e nt e r e d. Th e middl e war e provid e s th e acc e ss to 
these databases and e ncapsulates th e business logic around such transactions as when to 
acc e pt a bid, wh e n to d e clar e an it e m sold, e tc. In th e world of distribut e d computing, it 


- 57 - 



mak e s no diff e r e nc e whether th e "clients" using the application ar e e mploy e es of a singl e 
company or many Internet users throughout the world. H e r e in, exampl e s of applications 
und e r t e st will b e given, but they ar e not int e nd e d to imply limitations on th e us e of th e 
invention. The inventions d e scrib e d herein could be us e d by develop e rs of enterpri se 
wid e applications or w e b based applications. 

One advancement in th e N tiered model is that the middleware is very likely to b e 
compon e ntized and is very lik e ly to b e written to a compon e nt standard so that it will 
easily integrat e with softwar e at other tiers. Enterprise JavaB e ans (EJB) by Sun 
Microsystems, COM, DCOM and COM-f by Microsoft Corporation and CORB A by IBM 
are examples of component specification standards that are commercially avai labl e . 
H e rein, EJB is us e d as an e xample of a compon e nt standard us e d to impl e m e nt 
middleware in an N tiered model , but it should be appr e ciat e d that and th e concepts 
describ e d herein could b e us e d with oth e r compon e nt standards. 

BJBs ar c written in the JAVA language, which is intended to be "platform 
ind e p e nd e nt." Platform ind e pend e nt m e ans that an application is intended to perform th e 
same r e gardless of the hardware and operating s ystem on wfrich it is operating. Platform 
indep e ndenc e is achiev e d through the us e of a "container." A contain e r is softwar e that is 
designed for a sp e cific platform. It provides a standardized environm e nt that ensures th e 
application writt e n in th e platform ind e p e nd e nt language operates corr e ctly. The 
container is usually commercially available software and th e application d e v e loper will 
buy th e contain e r rath e r cr e at e it. 

Componentiz e d software is software that is d e signed to allow different pieces of 
the application, or "objects", to be created s e parately but still to have the obj e cts work 
tog e ther. For this to happen, the objects must hav e standard interfaces that can be 
und e rstood and acc e ssed by other objects. Some parts of th e s e int e rfaces are enforc e d by 
the software language. If th e interfac e s are not used, the s oftware obj e ct s will not b e able 
to work with oth e r obj e cts. Other practic e s are impos e d by convention. Usually, on e 
company has "control" over th e languag e and specifies programming practic e s that 
should b e follow e d by anyon e writing platform indep e nd e nt software in that languag e . 
B e caus e th e s e programming practices ar e known to everyone, the companies that create 
th e contain e rs can r e ly on th e m when cr e ating th e contain e r. As a result, if th e se 
practic e s are not followed, th e container might not operate properly. Thus, ther e is an 
indir e ct m e chanism for e nforcing th e s e practic e s. 

Typically, applications have b e en test e d in on e of two ways. The objects ar e 
t e st e d as they ar e writt e n. Each is t e st e d to e nsur e that it p e rforms the int e nded function. 
When th e objects are ass e mbl e d into a completed application, the entire application is 
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then usually test e d. H e r e tofor e , application t e sting has gen e rally b ee n don e by applying 
test inputs at the client end and observing the respons e of the application. There are 
s e veral shortcomings with this proc e ss. One is that it is relativ e ly labor intensiv e , 
particularly to dev e lop a load or scalability test. Th e re has b ee n no easy way to cr e ate 
th e t e st program, instantiat e it with test data, execute the test and aggr e gat e th e r e sults. 

Some tools, called "profilers have boon available. However, these tools track 
things such as disk usag e , m e mory usag e or thr e ad usag e of th e application und e r test. 
Th e y do not provide data about p e rformance of the application based on load. 

Other tools are availabl e to automat e th e e x e cution of tests on applications. For 
exampl e , RSW Software, Inc. of Waltham, MA, provides a product called c Load. This 
tool simulat e s load on an application und e r t e st and provides information about th e 
performance of th e application. How e ver, this tool does not provid e information about 
th e components in an application. W e hav e recognized that a softwar e d e veloper would 
find such information very us e ful 

Automatic t e st gen e ration tools, such as TestMast e r availabl e from T e radyn e 
Software and System T e st of Nashua, NH, are also available. Too ls of this type provide a 
m e ans to r e duc e th e manual e ffort of generating a t e st. TestMast e r works from a stat e 
mod e l of the application under t es t. — Such an application is very useful for gen e rating 
functional t e sts during the d e v e lopm e nt of an application. Onc e th e mod e l of th e 
application is s pecified, TestMaster can be instructed to g e n e rat e a suit e of tosts that can 
b e tailor e d for a particular task — such as to fully e x e rcis e som e portion of the application 
that has been chang e d. Mod e l bas e d t e sting is particularly useful for functional testing of 
larg e applications, but is not fully automatic b e cause it requir e s th e cr e ation of a stat e 
model of th e application being tested. 

W e hav e recogniz e d that a s e cond shortcoming of t e sting ent e rpris e wid e 
applications is the critical p e rformance criteria to mea s ure often relates to how th e 
application b e hav e s as th e number of simultan e ous us e rs incr e as e s. There ar e e xamples 
of websites crashing or operating so slow as to frustrate an ordinary user when too many 
users log on simultaneously. In th e past, load has b ee n simulat e d informally, such as by 
having s e veral peopl e try to use the application at the sam e time. Som e tools exist to 
provid e a load on an application for t e sting, such as e Load availabl e from RSW of 
Waltham, MA. 
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Howev e r, it has generally not b ee n until th e application is deploy e d into its int e nd e d 
operating e nvironment that th e performance of the application under load is known. 
Thus, th e bigg e st probl e m facing an application d e v e lop e r might not b e t e sting to s ee 
wh e ther e ach object perform s as design e d or e ven wh e th e r the objects work togeth e r as a 
system. H e r e tofore th e r e has b e en no available tool that will h e lp an application 
dev e loper asc e rtain how many simultan e ous users a middleware application can 
accommodat e giv e n a sp e cifi e d transaction r e spons e tim e or id e ntify which object in th e 
application, given real world load conditions, is causing the bottleneck.SUMMARY OF 

THE INVENTION 

With th e foregoing background in mind, it is an object of the invention to provide 

t e sting tools to facilitat e load based testing of N tiered applications. 
It is also an object to provid e automatic testing. 

Th e for e going and oth e r obj e cts ar e achi e v e d by a t e st syst e m that g e n e rat e s t e st 

cod e for components of an application under t e st using interface information about th e 
components. 

In th e pr e ferred embodim e nt, the generated test cod e simulates us e of a particular 

softwar e object within an application by a plurality of simultan e ous users. The numb e r 
of simultaneous users simulated is vari e d. 

In a pr e sently pr e f e rr e d e mbodim e nts, th e test cod e is g e n e rated from int e rfac e 
information for th e object under test, which id e ntifies methods of the object. Ranges and 
formats for data valu e s for th e inputs and outputs of th e m e thods ar e d e termin e d from 
properti e s specified in the int e rface information. Specific data values arc generated from 
us e r suppli e d tabl e s or using algorithms to pick allowable values within th e rang e . 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The inv e ntion will b e better understood by referenc e to the following more 

detail e d description and accompanying drawings in which 

FIG. 1 i s an illustration of an application und e r test by th e tost s ystem of th e 
inv e ntion; 

FIG. 2 is an illu s tration showing the test system of the invention in gr e ater detail; 
FIG. 3 is an illustration showing th e coordinator of FIG. 2 in great e r d e tail; 
FIG. 4 is a flow chart illustrating the proc e ss of coordinating execution of load 

FIG. 5 is an illustration showing th e cod e gen e rator of FIG. 2 in gr e ater detail; 
FIG. 6 illustrates a us e r int e rfac e of t e st syst e m 110 during a setup phas e ; 
FIG. 7 illustrates a us e r interface of t e st system 1 10 during th e sp e cification of a 
t e st cas e ; 

FIG. 8 illustrates a us e r interface of test system 110 during a different action as 

part of th e sp e cification of a t e st cas e ; 
FIG. 9 illustrates a user int e rface of test system 1 10 during a different action as 

part of th e sp e cification of a t e st case; 
FIG. 10 illustrates a u se r interfac e of test syst e m 1 10 during a different action as 

part of th e sp e cification of a t e st cas e ; 
FIG. 1 1 illustrates a user interfac e of test s y s tem 110 during a phase for reviewing 

the r e sults of a t e st cas e in tabular format; 
FIG. 12 illustrates a user interface of test system 1 10 during a phas e for revi e wing 

the results of a test cas e in graphical format; 
FIG. 13 illustrat e s a user interface of test system 110 during a phase for reviewing 

the results of a t e st cas e in an alt e rnative graphical format; and 
FIG. 11 illustrates a user interface of t es t system 1 10 during a phase for reviewing 

the r e sults of a t e st cas e in a tabular format. 
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DESCRIPTION OF THE PREFERRED EMBODIMENT 

FIG. 1 illustrates a test s ystem 110 according to th e present inv e ntion. The 

syst e m is t e sting application und e r test 114, Her e application und e r test 1 H is an 
application in th e N - tier e d model. More specifically, it is a three tiered database 
application. Application und e r t e st 1 1 4 could r e pr e s e nt a d atabas e for a bank or an 
insurance company or it could represent an Internet application. The specific function of 
application und e r t e st 1 H are not important to the inv e ntion. 

Also, the specific hardware on which t e st s ystem 1 10 and the application under 

t e st 114 r e sid e is not important to th e inv e ntion. It is suffici e nt if th e r e is som e 
conn e ction b e tw e en the two. In the illustration, that connection i s provid e d by network 
122. In the illustrat e d embodiment, it is cont e mplat e d that network 122 is part of a LAN 
operated by th e own e r of application und e r t e st 111, such as an Ethernet network. In this 
sc e nario, t e st syst e m 1 10 is install e d on a s e rver within th e LAN. Howev e r, many oth e r 
impl e mentations ar e possible. For example, network 122 could be a WAN owned by the 
owner of application under test 111. 

A further variation is that network 122 could the Int e rn e t. In that scenario, test 
syst e m 1 10 could b e located in a s e rv e r own e d by a testing company. 

A further possibility is that t e st system and application under test could be located 
in comput e rs owned by a t e sting company. Many applications ar e writt e n using platform 
indep e ndent t e chnology such that th e application will p e rform the sam e on many 
different platforms. Platform ind e p e ndent t e chnology is intended to mak e it e asy to run 
an application on any platform. Therefore, the application und e r test could be sent to a 
testing company, owning th e hardwar e to impl e m e nt test system 110, such as by 
uploading ov e r the Internet. Thereaft e r, the application und e r t e st could b e t e st e d as 
d e scrib e d h e r e in whil e running on a platform provid e d by th e t e sting company with the 
results of the test being downloaded over the Int e rn e t. 

Still other variations ar e possibl e . T e st syst e m 1 10 and application under t e st 11 A 
could physically be implemented in the sam e computer. However, that impl e mentation is 
not pr e s e ntly pr e ferr e d becaus e a singl e computer would have to b e v e ry larg e or would 
b e limited in th e siz e of applications that could b e test e d. Th e pres e ntly pref e rred 
embodim e nt us e s s e veral computers to impl e m e nt tost syst e m 1 10. 

Application und e r t e st 1 11 is a software application as known in the art. It 

includ e s middleware 116 that e ncapsulat e s som e business logic. A us e r access e s th e 
application through a client d e vic e . Many typ e s of client devices are possible, with the 
list growing as n e tworks b e com e mor e pr e val e nt. P e rsonal comput e rs, t e l e phon e syst e ms 
and ev e n hous e hold appliances with micro controll e rs could b e the client devic e . For 
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simplicity, th e cli e nt d e vic e is illustrated her e in as a p e rsonal comput e r (PC) 120, though 
th e specific typ e of client d e vice is not important to th e inv e ntion. PC 120 is connect e d 
to n e twork 122 and can th e r e for e acc e ss application under t e st 114. In us e , it is 
cont e mplated that there would b e multipl e users connected to application under test 111, 
but only one us e r is shown for simplicity. Th e numb e r of u se rs simultan e ously accessing 
application under test 1 11 is one indication of the "load" on th e application . 

Access to the application und e r t e st is, in th e illustrat e d e mbodiment, through 

Graphical User Interface (GUI) 1 24 of the type known in the art. Software to manag e 
int e ractions b e tw ee n multipl e users and an application is known. Such software is 
sometim e s called a web server. Web servers operat e in conjunction with a browser, 
which is software commonly found on most PC's. 

The web server and browser e xchange information in a standard format known as 

HTML. An HTML file contains tags, with sp e cific information associat e d with e ach tag. 
Th e tag signals, to the browser a typ e associated with the information, which allows th e 
brows e r to display th e information in on appropriat e format. For e xampl e , th e tag might 
signal wh e ther the information is a title for the page or whether the information is a link 
to anoth e r w e b pag e . Th e brows e r cr e at e s a scr e en display in a particular window 
running on PC 120 based on one or mor e HTML pages sent by the web s e rv e r. 

Wh e n a us e r inputs commands or data into th e window of th e brows e r, th e 

brows e r uses the information on the HTML page to format this information and send it to 
the w e b s e rver. In this way, the w e b server knows how to process th e commands and 
data that comes from th e us e r. 

GUI 121 pass e s the information and commands it r e c e iv e s on to middl e war e 1 16. 

In the exampl e of FIG. 1, middl e ware 1 16 is depicted as a middleware application 
cr e at e d with EJBs. Contain e rs 130 ar e , in a pr e f e rred embodim e nt, comm e rcially 
availabl e containers. Within a container, are numerous enterprise Java beans 132. Each 
Java b e an can mor e generally b e thought of as a component. GUI 121, bas e d on th e 
information from user PC 120, pas s es the information to the appropriate EJB 132. 
Outputs from application und e r t e st 1 11 ar e provid e d back through GUI 121 to PC 120 
for display to a us e r. 

EJB's 132, in the illustrated e xampl e , coll e ctiv e ly impl e ment a database 

application. EJB's 132 manage int e ractions with and process data from databases 126 
and 128. Th e y will perform such databas e functions as setting valu e s in a particular 
r e cord or g e tting values from a particular r e cord. Other functions ar e creating rows in th e 
databas e and finding rows in th e database. EJB's that acc e ss the databas e ar e oft e n 
ref e rr e d to as " e ntity beans." 
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Oth e r typ e s of EJB's p e rform computation or control functions. Th e s e ar e call e d 

"s e ssion beans." S e ssion beans p e rform such functions as validating data entries or 
r e porting to a user that an e ntry is e rron e ous. S e ssion b e ans g e n e rally call e ntity b e ans to 
perform database access. 

It will be appr e ciat e d that, whil e it is gen e rally pref e rable to se gr e gat e 
programming of the application in such a way that each typ e of databas e transaction i s 
controll e d by a singl e b e an that p e rforms only that function, som e entity b e ans will 
perform functions not strictly ti e d to databas e acc e ss. Lik e wis e , som e ses s ion beans will 
p e rform databas e acc e ss functions without calling an e ntity b e an. Thus, whil e diff e rent 
testing t e chniques will be d e scribed h e r e in for t e sting s e ssion beans and e ntity beans, it is 
possible that som e EJB's will hav e attribut e s of both e ntity and s e ssion b e ans. 
Cons e qu e ntly, a full test of any bean might employ techniqu es of te s ting entity beans and 
testing s e ssion b e ans. 

Test sy s tem 1 10 is able to acc e ss th e EJB's 1 32 of application und e r test 1 14 ov e r 
n e twork 122. In this way, e ach b e an can b e e x e rcised for testing. In th e pr e f e rr e d 
embodim e nt, th e tests are pr e dominately direct e d at determining the r e sponse tim e of the 
b e ans — or mor e g e n e rally d e t e rmining th e respons e tim e of compon e nts or obj e cts us e d 
to cr e at e the application und e r test. Knowing the r e sponse tim e of a b e an can allow 
conclusions about th e performanc e of an application. Th e d e tails of t e st syst e m 1 10 ar e 
described below. 

In th e illustrat e d e mbodiment, t e st syst e m 1 10 is softwar e install e d on on e or 
mor e serv e rs. It is conceptually much like application und e r t e st 114. In a pr e f e rred 
e mbodim e nt, t e st syst e m 1 10 is a JAVA application. Lik e application under t e st 1 H , t e st 
system 1 10 is controlled through a graphical us e r interfac e 150. GUI 150 might be a w e b 
s e rv e r as known in the art. On e or mor e application d e v e lop e rs or t e st e ngin ee rs might 
acces s test syst e m ov e r th e n e twork 122. In FIG. 1, PC's 152 and 154 are PC's us e d by 
t e st e rs who will control the t e st proc e ss. 

Like application under test 114, multipl e individual s might us e test system 1 1 0 
simultaneously. For e xampl e , multipl e test e rs might b e t e sting a single application. Each 
test e r might b e focused on testing diff e r e nt asp e cts of th e application. Alternatively, e ach 
test e r might b e t e sting a diff e r e nt application. Numerous applications might bo install e d 
on comput e rs on network 122. Test syst e m 1 10 might b e t e sting diff e rent applications at 
th e sam e tim e . 

Turning now to FIG. 2, d e tails of test syst e m 1 10 ar e shown. T e st system 1 10 
p e rforms s e v e ral functions. On e function is th e g e n e ration of t e st code. A second 
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function is to execut e th e t e st code to e x e rcis e on e or mor e EJB's in th e 
application under t e st. Another function is to r e cord and analyz e th e r e sults of 
executing th e test code. These functions are performed by software running on 
one or mor e comput e rs conn e cted to n e twork 122. Th e software is written using a 
comm e rcially available languag e to p e rform the functions described h e r e in. 

FIG. 2 s hows that test system 110 has a distribut e d archit e ctur e . Softwar e 
compon e nts ar e installed on s e v e ral diff e rent comput e rs. Multipl e comput e rs ar e us e d 
both to provid e capability for multipl e users, to a allow a us e r to perform multiple tasks 
and also to run v e ry larg e tests. Th e sp e cific numb e r of comput e rs and th e distribution of 
softwar e compon e nts of th e t e st syst e m on thos e computer s is not important to th e 
inv e ntion. 

Coordinator 21 0 is a softwar e application that interfaces with GUI 1 50. The main 
purpos e of coordinator 21 0 is to rout e us e r r e qu e sts to an appropriate s e rv e r in a fashion 
that i s transparent to a user. Turning to FIG. 3, coordinator 21 0 is shown in gr e at e r 
d e tail. It should be appr e ciat e d, though, that FIG. 3 shows th e conc e ptual structur e of 
coordinator 210. Coordinator 210 might not b e a singl e , s e parat e ly identified piec e of 
softwar e . It might, for e xampl e , b e impl e m e nt e d as coordination softwar e within th e 
various other components of test syst e m 1 10. Also, it s hould b e r e alized that a web 
s e rv e r us e d to impl e m e nt GUI 150 also provid e s coordination functions, such as qu e uing 
multiple requests from an individual or coordinating multiple us e rs. 

Coordinator 210 contains distribution unit 312. Distribution unit 3 12 is pr e f e rably 
a softwar e program running on a s e rver. As user requ e st s or e rec e iv e d from GUI 1 50, 
th e y ar e r e c e ived by distribution unit 312. As th e r e qu e sts ar e rec e iv e d, distribution unit 
312 d e termines th e type of resource ne e d e d to proc e ss th e r e quest. For example, a 
request to g e nerat e cod e must be sent to a s e rver that is running a cod e generator. 

Coordinator 210 includ e s s e veral queues to hold the pending requ e sts. Each 
qu e u e is impl e m e nt e d in th e m e mory of th e serv e r impl e m e nting coordinator 210. In 
FIG. 3, queue s 3 1 8A. . .3 1 8C arc illustrated. Each quou o 3 1 8A. . .3 18C corresponds to a 
particular typ e of r e sourc e . For e xampl e , qu e u e 318A could contain cod e g e n e rator 
r e qu e st s , queu e 3 1 8B could contain t e st engine requests and qu e ue 3 1 8C could contain 
data analysis r e qu e sts. Distribution unit s e nds e ach r e qu e st to on e of th e qu e u e s 
318A...318C, bas e d on th e type of r e sources n ee ded to process the request. 

Associat e d with e ach qu e u e 31 8 A...318C is qu e ue manag e r 320A...320C. Each 
qu e u e manag e r is preferably implem e nted as software running on th e s e rv e r 
impl e m e nting coordinator 210 or th e s e rv e r impl e m e nting th e r e l e vant piec e of 
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coordinator 210. Each qu e u e manag e r maintains a list of s e rv e rs within t e st syst e m 1 10 
that con respond to th e requ e sts in its associat e d queue. A queu e manager s e nds th e 
r e qu e st at th e top of th e qu e u e to a serv e r that is e quipp e d to handl e th e r e qu e st. Th e 
connection between the qu e ue manager and the serv e rs equipp e d to handle th e requests is 
ov e r n e twork 122. If th e r e ar e oth e r servers availabl e and still mor e requ e sts in th e 
queu e , th e qu e ue manag e r will send th e next r e quest in th e queu e to an availabl e server. 
Wh e n th e r e ar e no availabl e s e rv e rs, e ach queu e manag e r waits for on e of th e s e rv e rs to 
complete the proc e ssing of its assign e d r e qu e st. 

As th e r e quests ar e proc e ss e d, th e s e rv e rs, such as th e cod e g e n e rators and th e t e st 
engin e s report back to the qu e u e managers. In r e spons e , the queu e managers send 
anoth e r r e qu e st from th e qu e u e and also provid e th e r e sults back to th e distribution unit 
3 1 2. Distribution unit 312 can then reply back to the user that issued th e r e quest, 
indicating that th e r e qu e st was compl e t e d and e ith e r giving th e r e sults or giving th e 
location of the results. For example, after t e st cod e is g e n e rat e d, the u s er might r e ceive 
an indication of wh e r e th e t e st cod e is stored. Aft e r a t e st is e x e cut e d, the user might 
receiv e a report of the average e x e cution tim e for the test or th e location of a file stori ng 
e ach m e asur e m e nt mad e during th e t e st. 

It will be appreciat e d by on e of skill in th e art that softwar e systems that proc e s s 
us e r commands, including commands from multiple us e rs, ar e well known. Such 
systems must have an interfac e for receiving commands from a iis e r, proc e ssing those 
commands and pr e s e nting r e sults to the user. Such int e rfac e s also allow thos e r e sults to 
be us e d by th e us e r for implem e nting further commands. Such an int e rfac e is employ e d 
her e as well and is depicted g e n e rally as GUI 150. For e xampl e , GUI 150 will allow a 
user to enter a command that indicates code should be generated to t e st a particular 
application. Onc e the cod e is g e n e rat e d, GUI 1 50 allows th e us e r to sp e cify that a t e st 
should b e run u s ing that test cod e . 

It is possibl e that som e requ e sts will r e quire th e coordination of multiple hardwar e 
elem e nt s . As will b e d e scrib e d hereafter, one of the functions of the test e ngines is to 
apply a load that simulat e s multipl e us e rs. In som e instanc e s, one computer can simulate 
multipl e us e rs by running multiple client thr e ad s . How e ver, th e r e is a limit to the numb e r 
of cli e nt thr e ads that can run on a s e rv e r. 

Multiple threading the client t e st program has th e advantage of being able to 

gen e rat e v e ry larg e loads (t e ns of thousands to hundr e d of thousands virtual users). As 
such, multiple thr e ading th e cli e nt t e st program accurat e ly e mulat e s multiple users as far 
as th e softwar e component (EJB) d e ploy e d on th e application server is conc e rn e d. Only 


- 66 - 



on e sock e t is e stablish e d b e tw ee n th e host proc e ss and th e application s e rver and all th e 
threads communicat e with th e software obj e ct through the socket. 

Each t e st e ngin e can run on e or mor e Java Virtual Machin e s (JVMs), with each 
JVM having a resp e ctiv e socket to the application serv e r. A JVM is a softwar e 
impl e m e ntation of a proc e ssor d e sign e d to run Java cod e . Th e JVM runs within a 
process. Th e JVM acts as an interfac e b e twe e n compil e d Java binary cod e (byt e code) 
and th e microproc e ssor. Sinc e e ach JVM has a resp e ctiv e sock e t to the application 
s e rv e r, th e us e of J VM s v e ry accurately emulates a large user load on the application 
s e rv e r and on th e softwar e compon e nts. Each t e st e ngin e can run multipl e JVMs, and 
e ach J VM can run multipl e threads. Th e test e ngines can run any mixture of multipl e 
thr e ads and multiple JVMs. Th e test engin e s can also run a multithr e ad e d load from 
multiple process e s, applicabl e to testing EJBs and COM+ (Compon e nt Object Modul e s). 

For distribut e d testing, th e e ntire class fil e path specified by th e user is provid e d 
at th e remote machin e s. Th e entire class file path is compressed, also known as put into a 
jar, and th e compr e ss e d fil e or jar is s e nt to th e r e mot e machin e . In such a mann e r t e sting 
softwar e compon e nts with di s tributed e x e cution i s greatly simplifi e d. 

FIG. 4 shows the proc e ss by which qu e u e manag e r 320B coordinat e s th e actions 

of t e st e ngin e s located on s e parat e s e rvers. At step 4 10, queue manag e r 320B waits for 

th e r e quir e d numb e r of t e st e ngin e s to b e com e availabl e . Onc e th e t e st e ngin e s ar e 

availabl e , at st e p 4 12 qu e ue manag e r 320B s e nd s commands to each t e st e ngin e that will 

b e involv e d in th e t e st to download th e test cod e from th e appropriat e on e of th e cod e v 

g e n e rator s 21 2 A and 212B. 

Qu e ue manag e r 320B then begins th e process of s ynchronizing the t e st engin es 

located on diff e r e nt serv e rs. Various methods could be used to synchroniz e the 

serv e rs. For exampl e , if v e ry great accuracy is r e quired, each serv e r could b e 

equipped with radio rec e iv e rs that receiv e satellite transmissions that are intend e d 

to act as tim e r e f e r e nc e signals, such as in a GPS system. Calibration proc e ss e s 

could alt e rnatively b e used to det e rmin e th e amount of time it tak e s for a 

command to r e ach and be proc e ss e d by e ach s e rv e r. Commands could th e n b e 

s e nt to e ach s e rv e r at tim e s offset to mak e up for th e tim e diff e r e nc e s. In th e 

pr e ferred embodiment, a simpl e process is used. At st e p 4 1/1, qu e u e manag e r 

s e nds a m e ssag e to e ach s e rver to be acting as a test e ngine. Th e messag e a s k s 

that s e rv e r to report th e tim e as kept by its own int e rnal circuitry. 
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Th e multipl e JVMs can b e synchroniz e d by loading on ag e nt on each s yst e m th e 
JVMs* will run, then th e serv e r sends a message out to th e agents and r e qu es ts a time 
chock. Th e s e rv e r th e n s e nds th e cli e nt cod e to th e JVMs on th e r e mot e machin e s along 
with th e tim e at which th e JVMs should start e x e cuting the cli e nt code. In this mann e r 
highly accurat e and repeatabl e load g e neration is achi e v e d. Additionally, th e us e r can 
sp e cify a tim e offs e t for e ach of the machin e s and J VMs. For e xampl e , a first machine 
would be started; a s e cond machin e started t e n s e conds lat e r, e tc. Whil e this proc e ss is 
independent of the absolute tim e on each system, th e machine s could b e dir e cted to start 
at a pr e defin e d tim e bas e d on th e syst e m tim e . 

Additional synchronization techniqu e s includ e wh e re th e server s e nds th e cli e nt 
cod e to th e host, th e s e rv e r do e s a tim e ch e ck, th e n the s e rv e r sends th e start tim e to th e 
ho s t. Alternat e ly the server could s e nd the cli e nt cod e and start tim e to the hot. Y e t 
anoth e r synchronization techniqu e includ e s having a s e rvic e ( e .g. JINI) running on th e 
host computer. The s e rv e r sends a m e ssag e to synchronize in "X" number of ticks. Th e 
s e rvic e s wait for "X" numb e r of ticks th e n e x e cut e s th e cli e nt cod e . Furth e r, th e JVMs 
could run in an unsynchroniz e d mode, wherein th e J VMs start running the client cod e 
indep e nd e ntly of oth e r JVMs 

Upon receiving the int e rnal time as kept by each of th e servers, at step 416 queu e 
manag e r 320B adds th e sam e offs e t to e ach local tim e . At st e p 1 1 8, qu e u e manag e r 1 1 8 
send s the offs e t times back to the serv e rs. Th e offs e t local times b e com e the local starting 
time of th e t e st for e ach s e rv e r. Each s e rv e r is instruct e d to start the test wh e n its local 
time matches th e offs e t local time. In thi s way, all th e s e rv e rs start th e t e st 
simultan e ously. 

Qu e u e manag e r 320B waits for th e e x e cution of the t e st cod e at block 4 20. Som e 
t e st cas e s will e ntail multipl e t e sts. At st e p 122, a ch e ck is mad e of wheth e r th e particular 
test case being e xecuted requires mor e t e sts. For example, as will be described in great e r 
d e tail b e low, on e kind of t e st cas e that t e st syst e m 1 10 will run is a load test. During a 
load test, multipl e test e ngines, each e xecuting multiple client threads, execute to 
simulat e multipl e us e rs acc e ssing application und e r t e st 111. An operating param e t e r of 
application und e r test 1 1 4 is th e n m e asur e d. In the pref e rred e mbodim e nt, the numb e r of 
simultan e ous us e rs b e ing simulat e d can b e vari e d and th e operating param e t e r m e asur e d 
again. Talcing data at multipl e load conditions allows test system 1 10 to determin e the 
aff e ct of load on application und e r t e st 111. M e asur e m e nts of th e s e typ e s r e quir e that a 
full t e st cas e includ e multipl e repetitions of the same t e st with diff e r e nt load conditions. 

If th e r e ar e more conditions und e r which th e test should b e run, e x e cution 
proce e ds to st e p 124. At st e p 424, th e numb e r of cli e nt thr e ads is incr e as e d as needed for 
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th e n e w t e st cas e . Ex e cution th e n loops back to st e p 410. At st e p 410, queu e manag e r 
320B waits for the required numb e r of t e st e ngin e s to be available. When the hardwar e is 
availabl e , th e t e st is p e rform e d through st e ps 412,414,416,418 and 420 in th e sam e 
mann e r as for the first t e st condition. At st e p 4 22, a check is for whether the requ e st 
entails multiple t e st conditions. If th e r e ar e no furth e r t e st conditions, th e r e qu e st from 
queue 318B is consid e r e d complet e . If th e re ar e furth e r test conditions that need to b e 
run to compl e t e th e requ e st, th e proc e ss is again r e p e at e d. 

For t e st syst e m 1 10 to op e rat e , it is n e c e ssary that th e r e b e t es t code, A user could 
provid e t e st code. Or, t e st cod e could b e provid e d by automatic cod e g e neration systems, 
such as TESTMASTER sold by Teradyne Softwar e and Syst e m T e st of Nashua, NH. 
Howev e r, FIG. 2 illustrat e s that cod e g e n e rator s 212A and 212B ar e us e d in th e pr e f e rred 
embodiment to create the cod e . Turning to FIG, 5, great e r details of a code generator 212 
ar e shown. 

Cod e g e n e rator 212 contains s e v e ral scripts 512. Each s cript is a s e quence of 
st e ps that cod e g e n e rator 212 must p e rform to cr e at e cod e that p e rforms a c e rtain type of 
test. The scripts can be prepar e d in any conv e nient programming language. For each 
typ e of t e st that test syst e m 1 1 0 will p e rform, a script is provid e d. Us e r input on the type 
of t e st that is desired specifies which script 512 is to be us e d for g e n e rating cod e at any 
giv e n time. App e ndix 1 contains an e xampl e of a script. 

Th e s e l e ct e d script 512 ass e mbl e s t e st cod e 5 16. The information ne e ded to 
assembl e t e st cod e 516 com e s from s e v e ral sourc e s. On e sourc e of information is th e t e st 
template s 51 4 . Th e r e ar e som e steps that are ne e ded in almost any kind of test. For 
e xampl e , the obj e ct b e ing t e st e d must b e d e ploy e d and som e initialization s e qu e nc e is 
r e quir e d. If th e tests are timed, th e r e must be cod e that starts the test at a s p e cifi e d s tart 
time and an e nding tim e of th e t e st must b e r e cord e d. Also, th e r e must b e code that 
causes the requir e d data to be logged during the t e st. After th e test, there might also be 
som e t e rmination st e ps that ar e r e quir e d. For e xampl e , wh e r e th e initialization started 
with a request for a refer e nc e to a particular EJB, th e t e st cod e will likely terminat e with 
that r e f e r e nc e b e ing r e l e as e d. Th e t e st cod e to caus e th e s e st e ps to b e p e rform e d is 
captur e d in the se t of templat e s 51 4 . 

In addition, th e r e might b e diff e r e nt t e mplat e s to ensur e that th e t e st cod e 516 
appropriat e ly r e flect s inputs provided by th e user. For e xample, diff e rent containers 
might r e quir e differ e nt command fonnats to achi e v e th e sam e r e sult. On e way th e se 
diff e r e nt formats can b e r e flected in th e test code 516 is by having different templates for 
e ach contain e r. Alt e rnativ e ly, a us e r might b e abl e to sp e cify th e typ e of information that 
is to b e record e d during a test. In that instance, a data logging pref e r e nce might be 
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implem e nt e d by having a s e t of t e mplates that diff e r in th e command lin e s that caus e data 

to be recorded during a test. An e xample templat e is shown in Appendix 2. 

Th e templates are writt e n so that c e rtain spaces can b e fill e d in to customiz e th e 

code for the sp e cific object to be te s ted. In the preferred e mbodim e nt, code generator 

212 gen e rat e s cod e to test a specific EJB in an application und e r t e st. On e pi e c e of 

information that will need to be filled in for many t e mplates is a description of the EJB 

b e ing test e d. Anoth e r pi e c e of information that might b e includ e d is us e r cod e to put th e 

application und e r t e st in th e appropriat e s tat e for a t e st. For e xample, in t e sting a 

compon e nt of an application that manages a databas e of account information for a bank, 

it might be necessary to hav e a specific account created in the database to use for test 

purpo se s or it might oth e rwis e be n e c e ssary to initializ e an application b e for e t e sting it. 

The cod e n e ed e d to cause thes e ev e nts might be unique to th e application and will 

th e r e for e b e b e st ins e rt e d into th e t e st cod e by th e t e st e r testing th e application. In th e 

illustrat e d e mbodiment, this cod e is ins e rt e d into the t e mplat e and is then carri e d through 

to the final t e st cod e . 

The templat e might also contain spac e s for a human t e st e r to fill in oth e r 

information, such as specific data s e ts to us e for a t e st. How e v e r, in th e pr e s e ntly 

pr e f e rr e d e mbodim e nt, data s e ts ar e provid e d by th e human us e r in th e form of a 

data tabl e . 

DataTabl e s ar e us e d to apply larg e datas e ts to th e application und e r t e st. In ord e r 
to properly load t e st software components, it is necessary to sp e cify large datasets. Th e 
softwar e compon e nt is insp e ct e d and a templat e is provid e d. This t e mplat e may b e in a 
.CSV (comma s e parat e d valu e ) format although other formats could also be used. In the 
datatabl e th e columns ar e us e d for paramet e rs and th e rows r e pr e senting e ach us e r. Th e 
software cycles through e ach row of data as each user. If th e re ar e fewer rows than 
virtual cli e nts, th e softwar e will go to th e e nd of th e data s e t th e n start ov e r b e ginning 
with the first row. 

Cod e g e n e rator 212 could g e n e rat e functional t e sts. Functional t e sts ar e thos e 
t e sts that are dir e ct e d at d e t e rmining whether th e b e an correctly performs its r e quir e d 
limctions. In a functional t e st, th e softwar e und e r t e st is e x e rcis e d with many test cas e s to 
e nsur e that it op e rat e s correctly in e very state. Data tables indicating e xpected outputs 
for various inputs ar e us e d to creat e functional t e st softwar e . How e v e r, in th e pr e s e ntly 
pr e ferr e d e mbodim e nt, cod e generator 212 primarily g e n e rates test code that p e rforms 
load t e sts. In a load t e st, it is not n e c e ssary to stimulat e th e softwar e und e r t e st to 
e xercise ev e ry possible function and combination of functions the software is intended to 
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perform. Rath e r, it is usually suffici e nt to provid e on e t e st condition. The obj e ctiv e of 
the load test is to measur e how operation of th e software degrades as th e number of 
simultan e ous us e rs of th e application incr e as e s. 

In the preferred e mbodiment, t e st system 110 contains scripts 512 to impl e m e nt 
various types of load tests. On e typ e of load t e st determin e s r e spons e time of an EJB. 
Thi s allows th e t e st system to vary the load on the EJB and det e rmine degradation of 
r e spons e tim e in r e spons e to incr e as e d load. Anoth e r type of load t e st is a r e gr e ssion typ e 
load test. In a r e gr e ssion typ e test, th e script runs operations to d e t e rmine whether th e 
EJB r e sponds th e sam e way as it did to som e bas e lin e stimulus. In g e neral, the r e spons e 
to the bas e lin e stimulus r e pr e sents th e correct op e ration of the EJB. Having a regression 
typ e t e st allows th e t e st syst e m 1 10 to incr e as e th e load on a b e an and d e t e rmin e th e error 
rate as a function of load. 

To g e n e rat e t e st cod e 516 for th e s e typ e s of load t e sts, th e script 512 must cr e at e 
test cod e that is sp e cific to the b e an under t e st. Th e user provides information on which 
b e an to t e st through GUI 150. In th e pr e ferr e d e mbodiment, this information is provid e d 
by th e human test e r providing the nam e of the file within th e application under test that 
contains th e "d e ploym e nt descriptor" for th e sp e cific b e an und e r t e st. This information 
sp e cifies where in th e network to find th e bean under t e st. Script 512 us e s this 
information to asc e rtain what t e st cod e must b e g e n e rat e d to t e st th e b e an. 

Script 512 can generat e cod e by using th e attributes of the platform independ e nt 
languag e in which th e b e an is writt e n. For the exampl e of Sun JAVA languag e b e ing 
used h e re, each b e an ha s an application program interface call e d a "reflection." More 
particularly, e ach b e an has a "hom e " int e rfac e and a "r e mot e " int e rface. Th e "hom e " 
interface reveals information about the m e thods for creating or finding a remot e interfac e 
in th e b e an. Th e r e mot e int e rfac e r e v e als how this cod e can b e acc e ss e d from cli e nt 
software. Of particular int e r e st in the pr e ferr e d e mbodim e nt, tine home and remote 
int e rfac e s provid e th e information n ee d e d to cr e at e a t e st program to acc e ss th e b e an. 

AutoG e n creat e s tim e r fold e rs for each of the m e thods i n a test. The m e thods ar e 

organiz e d by topic (for exampl e all G e tt e r M e thods ar e put into th e G e t M e thod fold e r). 
Th e re s ults ar e displayed organiz e d in the sam e folder layout that AutoGon and the us e r 
d e fin e d in th e cli e nt program. This approach has th e advantag e that it is fl e xibl e and 
totally customizabl e by the us e r. Each parent folder calculat e s the average of th e 
r e spons e tim e of e ach it e m within th e folder and this works hi e rarchically. Th e par e nt 
fold e r may also p e rform oth e r calculations (such as total, e tc.) on the it e ms within e ach 
fold e r. 

Using th e r e flection, any program can determin e what ar e known as th e 


"prop e rti e s" and "m e thods" of a b e an. Th e prop e rti e s of a b e an d e scrib e th o data typ e s 
and attributes for a variabl e used in th e bean. Ev e ry variable used in th e bean must have 
a prop e rty associat e d with it. In this way, script 512 can automatically det e miin e what 
methods need to be exercised to test a bean and the variables that need to be generated in 
ord e r to provid e stimulus to th e m e thods. Th e variables that will b e by th e m e thods as 
they are test e d can also be d e termin e d. In the preferred embodiment, this information i s 
stor e d in symbol tabl e 515- 

Symbol table 51 5 is a file in any conv e ni e nt file format. Many format s for storing 
tabular data ar e known, such as .xml format. Onc e th e information on th e m e thods and 
prop e rti e s ar e captured in a tabl e , script 515 can use this information to create test code 
that e x e rcis e s th e m e thods and prop e rti e s of th e particular compon e nt und e r t e st. In 
particular, script 515 con automatically create a variable of th e corr e ct data type and 
assign it a valu e consistent with that typ e for any variabl e us e d in th e bean. 

FIG. 5 shows a data generator 518. Data generator 518 uses th e information 
d e rived from th e r e fl e ction int e rface to g e nerat e valu e s for variabl e s us e d during t e sting 
of a bean. There are many ways that appropriat e values could b e generated for each 
variabl e us e d in th e test of a particular b e an. How e v e r, in th e comm e rcial embodiment of 
the pres e nt invention, th e us e r is giv e n a choice of three different algorithms that data 
g e n e rator 518 will us e to g e n e rate data valu e s. Th e us e r can sp e cify "maximum," 
"minimum" or "random." If the maximum choice is specified, data generator 518 
analyz e s th e prop e rty d e scription obtain e d through the r e fl e ction interfac e and d e t e rmin e s 
the maximum permissibl e valu e . If the us e r specifies "minimum" then data generator 
518 g e n e rat e s th e small e st valu e possibl e . If th e us e r sp e cifi e s random, data gen e rator 
518 sel e cts a valu e at random betw e en the maximum and the minimum. 

In many instanc e s wher e a load t e st is desir e d, th e e xact valu e of a particular 
variable is not important. For e xample, when testing wh e th e r a bean can properly store 
and r e tri e v e a value from a databas e , it usually does not matter what valu e is stor e d and 
retriev e d. It only matters that the value that is read from th e database i s the same on e that 
was stored. Or, wh e n timing the op e ration of a particular b e an, it will oft e n not matt e r 
what valu e s are input to th e method. In th e se scenarios, data gen e rator 518 can 
automatically g e n e rat e th e valu e s for variabl e s used in th e t e st cod e . 

In cases where the specific values of th e variables us e d in a test are important, 
code gen e rator 212 provid e s th e us e r with anoth e r option. Rath e r than d e rive valu e s of 
variables from data g e n e rator 518, script 512 can b e instruct e d to d e riv e data values from 
a us e r provid e d data tabl e 520. A us e r might, for e xampl e , want to provid e a data table 
e ven for a load test wh e n the e xecution tim e of a particular function would depend on the 



valu e of th e input data. 

A data tabl e is impl e ment e d simply as a fil e on on e of the computers on network 

122. Th e e ntri e s in th e tabl e , specifying values for particular variables to us e as 

inputs and outputs to particular m e thods, are s e parated by d e limit e rs in the file. A 

standard format for such a tabl e is "comma s e parat e d valu e s" or CSV. In a 

pr e f e rr e d e mbodim e nt, t e st syst e m 1 10 includ e s a fil e editor — of th e typ e using 

conv e ntional t e chnology — for creating and e diting such a file. In addition, t es t 

system 1 10 would likely include the ability to import a fil e — again using known 

techniqu e s — that has th e r e quir e d format. 

Th e m e thods of a b e an d e scrib e the functions that b e an can perform. Part of th e 
d e scription of th e method is th e prop e rti e s of th e variabl e s that ar e inputs or outputs to th e 
method. A second part of the description of each method — which can also b e determined 
through the r e flection int e rfac e — is th e command n ee d e d to invok e this method. B e cause 
script 5 1 2 can d e t e rmine the cod e n ee ded to invok e any m e thod and, as d e scrib e d above, 
can g e n e rat e data valu e s suitabl e to provid e as inputs to that m e thod, script 512 can 
g e n e rate cod e to call any m e thod in the bean. 

In th e pr e f e rred e mbodim e nt, dir e ct e d at load t e sting, th e ord e r in which th e 
m e thods of a b e an are called is not critical to an e ffectiv e test. Thus, script 512 can 
automatically g e n e rat e us e ful t e st cod e by invoking e ach m e thod of th e bean. Th e ord e r 
in which the m e thods are invok e d do e s not matt e r if the only param e t e r that is m e asured 
is the tim e it tak e s th e m e thods to e x e cut e . 

More sophisticated t e sts can be automatically built by relying on th e pr e scrib e d 
patt e rn for th e languag e . In Sun JAVA, e ntity b e ans for controlling access to a databas e 
should have m e thods that have a prefix "s e t" or "get". Th e s e prefix e s signal that th e 
m e thod is e ith e r to writ e data into a databas e or to r e ad data from th e databas e . Th e 
suffix of the method nam e indicates which value is to be written or read in the databas e . 
For e xample, a m e thod nam e d s e tSSN should p e rform th e limction of writing into a 
database a valu e for a paramet e r id e ntified as SSN. A m e thod named g e tSSN should r e ad 
th e valu e from th e param e ter nam e d SSN. 

By taking advantag e of th e se pr e scrib e d patterns, script 512 can generate code to 
e x e rcis e and v e rify op e ration of both m e thods. A pi e ce of t e st code gen e rated to t e st 
th e s e m e thods would first ex e rcis e th e method s e tSSN by providing it an argum e nt 
cr e at e d by data g e n e rator 51 8 . Th e n, th e m e thod g e tSSN might b e e xercis e d. If th e g e t 
m e thod returns the sam e value as the argum e nt that was supplied to th e s e t m e thod, th e n 
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it can b e asc e rtain e d that th e databas e acc e ss e x e cut e d as e xp e ct e d. 

For many typ e s of enterpris e wide applications, th e beans most likely to be 
s e nsitiv e to load ar e thos e that acc e ss th e databas e . Thus, t e sting only s e t and g e t 
methods provides very us e ful load t e st information. 

How e v e r, th e amount of t e sting don e can b e e xpand e d wh e r e r e quir e d. Som e 
b e ans al s o contain m e thods that cr e ate or find rows in a databas e . By conv e ntion, 
m e thods that cr e at e or find rows in a databas e ar e nam e d starting with "cr e at e " or "find." 
Thus, by reflecting th e int e rface of th e bean, script 512 can also determine how to test 
thes e m e thods. Thes e m e thods can b e exercis e d similarly to the s e t and g e t m e thods. 
Th e properti e s r e v e aled through the application int e rfac e will described the format of 
e ach row in th e databas e . Thus, wh e n a cr e at e m e thod is us e d, data can b e automatically 
generated to fill that row, thereby fully ex e rcising th e cr e at e m e thod. 

In a pr e ferr e d e mbodim e nt, find m e thods are e xercis e d using data from a us e r 
supplied data tabl e 520. Often, databas e s have t e st rows ins e rt e d in them sp e cifically for 
testing. Such a t e st row would lik e ly b e writt e n into data tabl e 520. How e v e r, it would 
also b e possible to creat e a row, fill it with data and then e xercis e a find method to locate 
that row. 

Once th e commands that exercise the methods of an EJB are cr e ated, script 512 
can also ins e rt into th e cli e nt t e st cod e 516 th e commands that ar e n e c e ssary to record th e 
outputs of th e t e st. If a t e st is ch e cking for numbers of e rrors, then t e st code 516 needs to 
contain instructions that r e cord e rrors in log 216. Lik e wis e , if a t e st is m e asuring 
r e spons e time, th e t e st code 516 must contain instructions that write into log 216 the 
information from which respons e tim e can b e d e t e rmin e d. In the describ e d e mbodiment, 
all major database functions can be exercis e d with no us e r supplied test code. In som e 
instanc e s, it might b e possibl e to e x e rcise all th e functions with all t e st data automatically 
g e n e rat e d. All th e required information could b e generated from just the object code of 
th e application und e r t e st. An important featur e of th e pr e f e rr e d e mbodim e nt is that it is 
"minimally invasive" — m e aning that very little is r e quir e d of the user in order to conduct 
a t e st and th e t e st do e s not impact th e customer's e nvironm e nt. Ther e is no invasiv e t e st 
harn e ss. Th e cli e nt code runs e xactly lik e the cod e a user would writ e . 

In som e sc e narios, it will b e n e c e ssary or d e sirabl e for a user to insert sp e cific 
steps into th e t e st cod e 516. T e st system 1 1 0 allows for the user to e dit t o st code 5 16. 
For e xampl e , som e b e ans will p e rform calculations or p e rform functions oth e r than 
database acc e ss. In the se instanc e s, th e user might chos e to ins e rt test code that exercis e s 
th e s e functions. How e v e r, b e caus e bottl e necks in th e op e ration of many N ti e r e d 
applications occur in the e ntity beans that manag e databas e transactions, the simple 
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techniqu e s d e scrib e d her e in or e v e ry us e ful. 

App e ndix 3 gives an example of test code that has been generated. Onc e t e st code 
516 is gen e rat e d, with or without editing by a us e r, t e st syst e m 1 10 is r e ady to e x e cut e a 
tost. Us e r input to coordinator 210 trigg e rs the test. As d e scrib e d abov e , coordinator 210 
qu e u e s up r e qu e sts for t e sts and t e sts ar e e x e cut e d according to th e proc e ss pictur e d in 
FIG. 4, To simulat e on e us e r, the te s t code 516 is execut e d as a singl e thr e ad on on e of 
th e t e st e ngin e s 214A...21/IC. To simulat e multiple us e rs, multipl e thr e ads ar e initiat e d 
on on e or more of the t e st e ngin e s 214A...214C. 

Th e results of any t e sts ar e stor e d in log 216. FIG. 2 shows log 216 as a s e parat e 
hardware el e m e nt attached to n e twork 122. Log 216 could be any typ e of storag e 
traditionally found in a network, such as a tape or disk driv e attached to a computer 
sewer. For simplicity, it is shown as a separate unit, but could easily be part of any other 
serv e r in the network. 

Many types of data could be stor e d in log 216. For example, th e re are sev e ral 
possible ways that "r e sponse tim e " could b e m e asur e d. On e way is that th e total tim e to 
ex e cut e all the methods in a bean could be m e asur e d. Anoth e r way i s that th e start up 
tim e of a b e an could b e m e asur e d. Th e startup tim e is th e tim e it tak e s from when th e 
b e an is first access e d until the first method is abl e to execute. Another way to m e asur e 
r e spons e tim e is to m e asur e th e tim e it tak e s for e ach method to e x e cut e . As another 
variation, r e sponse time could b e m e asur e d bas e d on how long it takes just the "g e t " 
m e thods to e x e cut e or just th e "set " m e thods to e x e cut e . 

Differ e nt m e asurem e nts must b e r e cord e d, depending on which measure of 
r e spons e tim e is used. For e xampl e , if only th e total r e spons e tim e is r e quir e d, it is 
s uffici e nt if the test code simply records th e time that the portion of th e test cod e that 
e x e rcis e s all th e m e thods starts and stops ex e cuting. If th e startup respons e tim e is 
r e quir e d, then th e client test code must r e cord the time that it first acc e sses the bean under 
t e st and th e tim e wh e n th e first m e thod in th e t e st sequ e nc e is r e ady to b e call e d. On th e 
other hand, if th e r e spons e tim e is going to be computed for e ach m e thod, the cli e nt t e st 
cod e must r e cord the tim e befor e and aft e r it calls e ach method and som e indication of 
the method being called must also be logged. Similar information must b e record e d if 
r e spons e s of just "g e t " or "s e t " functions ar e to b e measur e d, though the infonnation 
needs to be recorded for only a subset of th e m e thods in th e se cas e s. 

In addition, wh e n ther e ar e multiple us e rs b e ing simulat e d, ther e ar e multipl e 
valu e s for e ach data point. For exampl e , if t e st system 1 10 is simulating 100 us e rs, th e 
tim e that it tak e s for th e b e an to respond to each simulat e d us e r could b e diff e r e nt, 
leading to up to 1 00 different measur e m e nts of r e spons e tim e . The r e spons e time for 1 00 
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us e rs could b e pr e s e nt e d as th e maximum r e spons e time, i. e . th e tim e it takes for all 100 
simulat e d us e rs to finish e x e rcising th e b e an under test. Alt e rnativ e , the averag e time to 
compl e t e could b e r e port e d as th e response tim e . As anoth e r variation, th e rang e of 
valu e s could be report e d. 

In th e pr e f e rr e d e mbodim e nt, th e cli e nt t e st cod e 516 contains the instructions that 
r e cord all of th e information that would b e n ee ded for any possibl e m e asur e of r e spons e 
tim e and e v e ry possibl e display format. The time is r e cord e d b e for e and aft e r th e 
e x e cution of e v e ry m e thod. Also, an indication that allows th e m e thod to b e identified is 
r e cord e d in log 216. To support analysis bas e d on factors other than delay, th e actual and 
e xp e cted results of the execution of e ach m e thod ar e r e cord e d s o that errors can b e 
d e t e ct e d. Also, the occurr e nc e s of exc e ptions ar e also r e cord e d in th e log. Th e n, data 
analyzer 218 can review th e log and display the respons e time according to any format 
and using any d e finition of respons e tim e d e sir e d. Or th e data analyz e r can count the 
numb e r of exceptions or e rrors. 

Onc e th e data is stor e d, th e us e r can sp e cify th e desir e d format in which th e data 
is to b e pres e nt e d. Data analyzer 218 acc e pts commands from the tester conc e rning th e 
format of th e output and analyzes th e data appropriat e ly. In a pr e f e rr e d embodim e nt, t e st 
syst e m 1 10 has th e ability to pr e sent the results of t e sts graphically to aid the tester in 
und e rstanding th e op e rations — particularly p e rformanc e bottl e n e ck — of application und e r 
test 111. 

Data analyz e r 218 g e n e rat es output in s e v e ral us e ful formats. On e important 
output is a response time v e rsus load graph. Log fil e 216 contains the starting and 
stopping tim e s of e x e cution t e sts for a particular t e st cas e . Th e test cas e includes th e 
same m e asurements at s e v e ral diff e rent load conditions (i. e . with the test e ngines 
214A...2HC simulating diff e r e nt numb e rs of simultaneous us e rs). Thus, data analyzer 
can r e ad through the data in log 216 and id e ntify results obtained at differ e nt load 
conditions. This data can b e graph e d. 

Another useful analysis is th e number of errors per s e cond that are generat e d as a 
function of the number of simultan e ous us e rs. To perform this analysis, t e st cod e 516 
could contain instructions that writ e an e rror m e ssag e into log 21 6 when e v e r a t e st 
stat e m e nt produc e s an incorr e ct r e sult. In th e databas e cont e xt, incorr e ct r e sults could be 
id e ntified when the "g e t" function do e s not r e turn th e sam e valu e as was pass e d as an 
argum e nt to th e "s e t" function. Or, errors might b e id e ntifi e d when a b e an, wh e n 
access e d, does not r e spond or responds with an exception condition. As abov e , data 
analyz e r 21 8 can pass through th e log fil e , r e ading th e numb e rs of e rrors at diff e r e nt 
simulat e d load conditions. If d e sir e d, the errors can b e e xpr e ss e d as an e rror count, or as 
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an e rror rat e by dividing th e error count by th e tim e it took for th e t e st to run. 

Some examples of the types of outputs that might be provided arc graphs 
showing: transactions per second versus number of users; response time versus number 
of users; exceptions versus numbers of users; errors versus numbers of users; response 
time by method; response time versus run time and transactions per second versus run 
time. Different ways to measure response time were discussed above. In the preferred 
embodiment, a transaction is defined as the execution of one method, though other 
definitions arc possible. 

Run time is defined as the total elapsed time in running the test case, and would 
include the time to set up the execution of EJBs. Viewing response time as a function of 
elapsed time is useful, for example, in revealing problems such as u memory leaks". A 
memory leak refers to a condition in which portions of the memory on the server running 
the application under test gets used for unproductive things. As more memory is used 
unproduct ively, there is less memory available for running the application under test and 
execution slows over time. — Alternatively, viewing results in this format might reveal that 
the application under test is effectively utilizing caching. If caching is used effectively, 
the execution time might decrease as elapsed time increases. Turning now to FIG. 6, 
op e ration of test syst e m 1 1 0 from th e t e st e r's perspectiv e is illustrat e d. FIG.6 illustrat e s 
th e t e st e r int e rface to be a traditional w e b brows e r int e rfac e as it would app e ar on th e 
t e rminal of PC 1 52 or 1 5 4 . This typ e of int e rfac e is the r e sult of using a web s e rv e r to 
impl e m e nt GUI 150 with comm e rcially availabl e browser software on th e t e ster PCs 152 
and 15 4 . 

El e m e nt 612 is th e brows e r toolbar. Th e brows e r provid e s th e ability for th e 
t e st e r to p e rform such functions as printing and conn e cting to pages pr e s e nted by various 
serv e rs in th e syst e m. Th e t e st e r p e rforms th e se functions through th e us e of th e tool bar 
612. Fi e ld 614 indicates th e s erver to which the PC 152 or 154 is currently connect e d. In 
th e illustrated e xampl e , fi e ld 614 contains th e addr e ss on network 122 for th e s e rver 
containing GUI 150. 

Window 610 is th e ar e a of th e scr e en of PC 152 or 151 wh e r e information 
sp e cific to test system 1 10 is displayed. As with traditional web bas e d applications, thi s 
information is display e d as th e r e sult of HTML files that ar e download e d ov e r n e twork 
1 22. C e rtain el e ment s displayed in window 610 repres e nt hyp e rlinks, which when 
acc e ss e d by a t e st e r caus e oth e r pag e s to b e download e d so that diff e r e nt information is 
displayed for the t e ster. Oth e r e l e ments displayed in window 610 represent data entry 
fi e lds. Wh e n a human t e st e r fills in th e s e fi e lds, information is submitt e d to t e st syst e m , 
4-4-Or 
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Tabs 616 repr e s e nt choic e s a t e st e r con s e l e ct for op e rating mod e , FIG, 6 s hows 
three tabs. There is on e tab for "setup", on e for "t e st cas e " and on e for "r e sults". Each of 
th e tabs is hyp e rlink e d. Th e se hyp e rlinks conn e ct to pag e s on s e rv e rs in th e n e twork that 
are programm e d to manage the t e st syst e m through a particular phase. S e tup is for 
configuring th e t e st syst e m, such as by id e nti lying addr e sses for servers on n e twork 122 
that contain test engines. "Test cas e " is for creating and running a particular test cas e , 
which in th e pr e f e rr e d e mbodim e nt m e an s t es t cod e for a particular b e an and param e t e rs 
specifying how that test code is to run. "R e sults" is for viewing the results of a t e st cas e 
that has e x e cut e d. In FIG. 6, th e "SETUP" tab is shown s e l e ct e d, m e aning that th e 
balance of window 610 displays hyp e rlinks and fields that are appropriate for entering 
data or commands for th e SETUP phas e . 

R e gion 618 contains a set of hyp e rlinks. Thes e hyp e rlinks pres e nt a list of 
choices to the us e r about what can b e s e tup. Sel e cting on e of these hyp e rlinks will 
chang e th e information app e aring in r e gion 620. It i s well known in th e art of w e b bas e d 
softwar e to hav e hyp e rlinks on on e part of window chang e information app e aring in 
another part of th e window. 

In th e Exampl e of FIG. 6, th e hyp e rlink "Machin e " in r e gion 618 has b e en 
sel e ct e d. Ther e for e , region 620 contains fields that can b e us e d to id e ntify a machine to 
b e add e d to t e st syst e m 110. In FIG. 6, a scr ee n to add a cli e nt host comput e r is shown. 
A cli e nt host comput e r acts as a t e st e ngine 21 4 A, . .21 4C. 

R e gion 61 8 also contains hyperlinks for "Proj e cts" and "Data Tables". As 
describ e d abov e , test system 1 1 0 can b e used by multiple test e rs or can be us e d to test 
multipl e applications und e r t e st. To segr e gat e th e information relating to various tasks, a 
"Proj e ct" is d e fin e d. All information r e lating to a particular proj e ct is id e ntified with th e 
proj e ct. In this way, t e st system 1 10 can id e ntify information that is logically r e lat e d. 
For e xampl e , test code dev e lop e d to t e st a particular b e an in an application will be given 
th e sam e proj e ct nam e as t e st cod e to t e st oth e r b e ans in that application. In this way, 
general information about th e application — such a s the path to th e s e rv e r through which 
th e application can b e acc e ss e d is stor e d onc e with th e proj e ct rather than with each pi e c e 
of test cod e . As another exampl e , th e test r e sults can b e associated with the t e st cod e that 
g e n e rat e d th e m through th e us e of proj e cts. 

The hyp e rlink for "Data Tables" allows th e t e st e r to identify to the system wher e 
data tabl e s ar e stor e d to t e st sp e cific b e ans. In g e n e ral, th e t e st e r will creat e th e data 
tables by hand or automatically apart from t e st syst e m 110 based on th e features that the 
t e ster desir e s to hav e t e st e d. Th e data tabl e s will b e stor e d in fil e s on a comput e r on 
network 122. B e fore use by test system 1 10, the t e st e r must provid e the network addr e ss 


- 78 - 



for that fil e . 

Fi e ld 622 is a menu fi e ld. It is a drop down menu box. Wh e n acc ess ed, it will 
display a m e nu of choic e s of th e actions th e test e r can sp e cify for t e st system 1 10, Th e 
contents of the m e nu is cont e xt s e nsitive, m e aning that for any gi v e n page, only the menu 
choices that ar e appropriat e ar e display e d. Actions that a user might want to choos e from 
the menu are things such as edit, d e l e t e , add new, rename, e tc. 

Turning now to FIG. 7, a scr ee n of t e st e r PC 152 or 151 is shown wh e n th e TEST 
CASE tab s e lected. Selecting the TEST CASE tab allows the t e ster to specify what is to 
b e t e st e d and how th e t e st is to b e conduct e d. In th e e xampl e of FIG. 7, window 610 
contains information that d e scribes a test cas e . This particular page is display e d wh e n the 
" e dit" has b e en s e l e ct e d in m e nu fi e ld 622. Fi e ld 710 indicat e s th e nam e of th e proj e ct 
to which the t e st case is associated. Fi e ld 710 is a scr e en display elem e nt known as a 
drop down box. Wh e n activat e d by a t e ster, fi e ld 710 will b e com e a list of all proj e cts 
that have be e n pr e viously created by and tester using t e st system 110. As shown in FIG. 
7, a proj e ct call e d "Demo Proj e ct" is b e ing us e d. 

Field 712 id e ntifie s the particular test cas e by nam e . Field 712 is also a drop 
down box, allowing pr e viously d e fin e d t e st cas e s to b e s e l e ct e d from a list. In addition, 
one of the options that app e ars when th e drop down box is acce s s e d i s "<New Test 
Case>". Wh e n s e l e cted, a n e w t e st cas e is creat e d and information about this t e st cas e 
can be ent e red. This typ e of menu is common for programs with graphical user interfaces 
and is us e d at s e v e ral points in the int e rfac e for pr e s e nting choices to a human t e st e r. 

In the exampl e of FIG. 7, a test case "Customer Test" has been creat e d and 
information has alr e ady b ee n e nt e red. In r e gion 714, th e r e ar e a s e ri e s of fi e lds in which 
data can be entered or changed to define or modify the test case. In fi e ld 71 6, a name can 
b e provid e d for th e test cas e . This nam e allows th e test cas e to b e ref e renced in oth e r 
parts of the t e st syst e m. 

In fi e ld 718, a d e scription of th e t e st case can b e e nter e d. This description is not 
used by the automatic featur e s of th e test syst e m , but can act as a note to a human t e ster 
to signify what th e t e st cas e does or why it was cr e at e d. Fi e ld 720 is lik e wis e not used by 
th e automated system. However, this fi e ld holds th e nam e of an individual who authored 
th e t e st cas e to facilitate oth e r administrativ e functions. 

Field 722 is a "radio button" typ e fi e ld. It presents a tester with a list of choices, 
only on e of which can b e s e l e ct e d at a tim e . In this e xampl e , fi e ld 722 allows th e t e st e r 
to sp e cify the type of test. As previously described, code g e n e rators 212A and 212B 
contain a plurality of scripts 512 that g e n e rate tost cod e . As described abov e , th e script 
as se mbles templat e s and gen e rate command lines for a particular type of test that is to b e 
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conduct e d. Thus, th e t e st e r must sp e cify th e t e st type in ord e r to allow th e appropriat e 

script to be select e d. In this exampl e , only two typ e s of tests ar e pr e sented as options to a 

t e ster — a load t e st and a functional test. Th e s e typ e s of t e sts w e r e discuss e d abov e . Fi e ld 

724 allow s th e t e st e r to sp e cify a "deployment descriptor." In th e J AVA language, ev e ry 

b e an has associat e d with it a d e ploym e nt d e scriptor. Th e d e ploym e nt d e scriptor is a fil e 

that identifies the b e an and d e fines the paramet e rs with which th e EJB will be deploy e d 

on th e applications s e rv e r. Exampl e s of th e param e t e rs ar e th e numb e r of instantiations 

of the EJB with which to start the applications s e rv e r (som e times call e d a "pooling 

numb e r") and how much memory is allocat e d to th e b e an. Thes e functions ar e p e rform e d 

by th e container 130. 

Th e t e st e r provid e s th e d e ploym e nt d e scriptor by e nt e ring the path to a fil e on 

n e twork 122. The test s yst e m 1 10 reads th e deploym e nt descriptor to find the nam e of 

th e b e an und e r test and th e n to acc e ss th e b e an through r e flection to d e t e rmin e its 

m e thods and properti e s. 

Field 726 allows th e t e st e r to sp e cify the type of data to be used in cr e ating the 

t e st cod e 516. As d e scrib e d above, in the preferred e mbodiment, the data can b e 

automatically gen e rat e d by te s t system 1 10 or can b e suppli e d through a data 

table. For automatic data g e neration, th e data can be g e nerat e d by using th e 

maximum possibl e valu e of e ach variabl e , the minimum possible valu e or a valu e 

randomly se lect e d b e tw ee n th e maximum and th e minimum. Alt e rnativ e ly, th e 

data can b e sp e cifi e d by a data tabl e . In FIG. 7, fi e ld 726 indicates that t e st e r 

d e sir e s t e st syst e m 1 10 to g e n e rat e data using th e data tabl e named dd.csv. If th e 

t e st e r had want e d th e t e st system to automatically g e nerate data, th e t e ster would 

sp e cify in field 726 wh e th e r th e data was to be g e n e rat e d randomly or whether the 

maximum or minimum valu e s wer e to b e iis e d. 

Field 728 allows th e user to sp e cify th e server on which the application under test 
ains. FIG. 1 show s that th e b e ans 132 of th e application under t e st ar e within a contain e r 
1 30. In this cont e xt, "s e rv e r" r e f e rs to th e software that creates the contain e r for th e 
application. For e ach platform ind e p e nd e nt languag e , th e r e ar e a limit e d numb e r of 
comm e rcially availabl e s e rv e rs. T e st syst e m 1 10 contains script fil e s that will g e n e rat e 
th e appropriat e t e st cod e for any s e rv e r. While most of th e cli e nt t e st cod e will b e s e rv e r 
independ e nt, it is possibl e that s e rv e rs will impl e m e nt certain functions in uniqu e ways. 
Thus, th e re n ee ds to b e script fil e s that account for the functions that ar e impl e m e nt e d 
diff e rently on c e rtain s e rv e rs. 
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FIG. 8 shows a scr ee n display wh e n th e t e st e r has us e d m e nu field 622 and 
sel e cted to hav e the deployment d e scriptor for a test case to b e di s play e d. If desir e d, th e 
tester can th e n e dit th e d e ploym e nt d e scriptor to try alt e rnativ e configurations. 

FIG. 9 shows a scre e n display when a test e r has sel e cted from menu field 622 to 
have th e g e nerated t e st cod e display e d. In FIG. 9, th e t e st code 516 is id e ntifi e d as 
"cli e nt code" b e cause it will simulat e the operation of a cli e nt 120 of th e application 
und e r test 111. Th e display e d cod e corr e sponds to th e cod e g e n e rat e d for th e proj e ct and 
t e st cas e id e ntified in fi e lds 71 0 and 712. In this scr e en, the test e r can also edit th e test 
cod e. 

One instanc e wh e n a t e ster might d e sir e to edit t e st cod e is when most of th e 
commands in th e t e st cod e can b e automatically g e nerat e d, but c e rtain commands must 
hav e sp e cific data valu e s for th e application under t e st to function properly. Th e test e r 
could hav e test syst e m 110 automatically g e nerat e th e t e st cod e , but th e n manually e dit 
th e f e w data valu e s that matter. As an exampl e , a particular b e an might includ e a m e thod 
that proc e ss e s positiv e and n e gative valu e s diff e r e ntly. Th e t e st e r might know that 
processing n e gativ e numb e rs tak e s long e r and ther e fore change a randomly gen e rat e d 
valu e to a n e gativ e numb e r. 

An alternativ e sc e nario in which a t e ster might wish to edit test code 5 16 is wh e n 
th e b e an b e ing t e st e d contains m e thods to b e t e st e d oth e r than thos e that follow a patt e rn, 
such as the "s e t", "g e t", "cr e at e 1 " and "find" methods describ e d above. The t e ster might 
cr e at e t e st cod e that t e sts m e thods that do not follow th e patt e rn. This code could th e n b e 
insert e d by the human t e st e r into the g e n e rated test code. 

FIG. 1 0 shows a scr ee n display for anoth e r function p e rform e d by a human t e st e r 
whil e specifying the TEST CASE, also selected through menu field 622. FIG. 10 is a 
scr e en display us e d wh e n th e t e st cas e is to b e run. Th e project and specific t e st cas e is 
sp e cifi e d by e ntries in fields 710 and 712. Information about how to run tho test case is 
e nt e r e d in r e gion 1010. 

Region 1010 contains s e v e ral fields. Field 1012 indicates the nam e of the file in 
log 216 where th e r e sults of e x e cuting th e t e st cas e will be stor e d. In this way, data 
analyz e r 21 8 will b e abl e to locate th e data for a particular t e st cas e to analyze and 
pr e s e nt to a t e st e r in th e d e sir e d form. 

Field 1014 giv e s th e URL — or network addr e ss — for the application under t e st. 
This information could b e id e ntifi e d by using th e nam e for a particular machine that was 
previously set - up by the human t e st e r. Field 1016 giv e s the URL — or network address 
for a s e rv e r to us e as a t e st engin e . Again, th e s e rv e r could b e id e ntifi e d by using the 
nam e for a particular machine that was previously set up by the human tester. Th e scr ee n 


- 81 - 


display e d in FIG . 10 is us e d for an e mbodim e nt of th e inv e ntion wh e r e all t e st 
simultaneously executing copi e s of the client t e st code are run on a single machine. If 
t e st syst e m 1 10 includes multipl e t e st e ngin e s and automatically schedul e s e xecution of 
test cod e as describ e d in conjunction with FIG. 4 above, then field 1016 is not required. 

Fi e ld 1018 giv e s th e maximum numb e r of concurrent us e rs to b e simulated at on e 
time during the ex e cution of the t e st case. Field 1020 allows th e user to speci fy th e rat e at 
which th e numb e r of simultan e ous us e rs will b e simulat e d during a t e st. In the e xampl e of 
FIG. 10, the test case will be compl e ted after 100 users have been simulated 
simultan e ously and th e numb e r of simultan e ous us e rs will incr e as e by 10 each tim e th e 
test code is run. For th e e xamples giv e n here, 10 copi e s of th e cli e nt test code shown in 
FIG. 9 will first b e ex e cut e d simultan e ously. Th e n 20 copi e s of that t e st code will b e 
executed simultaneously. Th e t e st cod e will be repeated for 30, 4 0, 50, 60, 70, 80, 90 and 
100 simultan e ous users befor e th e t e st cas e is compl e t e d. 

After the t e st case has b ee n run, the tester can view and analyz e the results. Th e 
human t e st e r can acc e ss the functions of th e t e st syst e m 110 that display and analyz e 
result s by sel e cting th e RESULTS tab from tabs 616. FIG. 1 1 shows a page displayed 
wh e n th e RESULTS tab is s e l e cted. The pag e shown in FIG. 1 1 is for wh e n th e test e r has 
r e qu e st e d to se e summary data through use of menu field 622. 

Fi e lds 710 and 712 ar e also pr e s e nt on the pag e shown in FIG. 1 1. This 
information is us e d by the test syst e m to locate th e appropriat e data to display. In 
addition, th e r e is a field 1110 that allows th e us e r to ent e r the name of th e r e sults file to 
display. As described in conjunction with FIG. 10, the user specifies in field 1012 th e 
nam e of a r e sults fil e to hold th e r e sults of a particular run of a t e st cas e . Th e nam e of the 
r e sults fil e for d e sir e d run of th e t e st is e ntered in filed 1110. 

FIG. 1 1 shows that window 610 contains a r e gion 1112 that lists th e r e sults of a 
run of a particular t e st case in summary fashion. Part of the information in region 1112 is 
simply a display of information input by th e t e st e r wh e n e diting or running th e t e st cas e . 
For exampl e , the targ e t container, or the contain e r 130 of application under t e st 1 14 is 
list e d. Th e maximum numb e r of concurr e nt us e rs simulated during th e t e st is also 
displayed. Th e fil e containing the t e st data use for the run of the test case that g e n e rated 
the r e sult s is also display e d as is th e deployment d e scriptor. Thes e valu e s are display e d 
for eas e of us e by the human tester. 

R e gion 1112 also includes information that was d e v e lop e d by data analyz e r 218 
from th e data gather e d for th e s p e cifi e d run of the test case. In FIG. 1 1, th e pi e c e s of 
information that are display e d ar e th e av e rag e r e spons e tim e and th e maximum r e spons e 
tim e . As describ e d abov e , as a t e st cas e runs, th e s tart and stop times of th e ex e cution of 
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th e t e st cod e is r e cord e d in log 216. Wh e n th e t e st ood e is run multiple tim e s, e ach tim e 
simulating a differ e nt number of users, the start and stop time is r e corded for e ach 
numb e r of us e rs. Data analyzer can d e t e rmin e the r e spons e tim e by simply comp uting 
the differenc e b e tween the start and stop times. Once valu e s are d e t e rmined, they can b e 
av e rag e d or th e maximum can b e identifi e d, 

FIG. 1 2 shows a different pag e that can b e s e l e ct e d by the user to s ee the results in 
graphical format. As abov e , fi e lds 710, 712 and 1110 allow th e us e r to sp e cify which run 
of a particular test case is us e d to creat e the results. Th e graphical display in FIG. 12 is a 
graph showing numb e rs of transactions p e r s e cond as th e d e p e nd e nt variabl e with th e 
number of simultaneous us e rs as th e ind e p e ndent variabl e . As d e scribed abov e , th e 
information n ee d e d to comput e the data valu e s for this graph is stor e d in log 2 16 aft e r th e 
t e st cas e is run and data analyzer 218 can r e tri e ve it. For creation of the graph in FIG. 12, 
transactions p e r s e cond was d e fin e d as th e av e rag e numb e r of m e thods ex e cut e d p e r 
second p e r user. This valu e is essentially the reciprocal of r e sponse time. 

FIG. 13 shows a scr ee n us e ful for displaying r e sults to a human t e st e r in a slightly 
diff e r e nt embodim e nt. As with FIG. 12, th e scr ee n display in FIG. 12 is acc e ssed wh e n 
the "RESULTS" tab is s e l e cted. Also lik e in FIG. 12, the page shown in FIG. 13 
includ e s fi e lds 710, 712 and 1110 that allow th e human test e r to specify which r e sults ar e 
to b e display e d. 

Th e page shown in FIG. 13 includes an alt e rnativ e way for the user to sp e cify th e 
format of th e display. Th e scr e en in FIG. 13 includ e s m e nu fi e lds 1310 and 1312. Menu 
field 1310 allows th e t e ster to s pecify the manner in which respons e time is to b e 
calculat e d. In FIG. 13, a valu e of "total" has b ee n s e l e ct e d in fi e ld 1310. As d e scrib e d 
above, the "total" r e sponse time is m e asur e d as th e time from first access of a b e an until 
all m e thods of th e bean hav e b ee n e x e rcis e d. Oth e r choic e s in m e nu fi e ld 1310 allow a 
t e st e r to sp e cify that result be display e d for diff e rent measures of r e spons e time. As 
d e scrib e d abov e , th e pr e sently pr e f e rr e d e mbodim e nt can m e asur e response tim e just on 
th e start up time or r e spons e time for individual methods or for get functions and set 
functions. 

Fi e ld 1312 allows a user to sp e cify the format of th e display. In FIG. 13, the 
display is in HiLo format. In this format, r e sults ar e display e d as bars, such as bar 1316. 
Each bar spans from th e fast e st respons e tim e to th e slowest response time. A tick mark 
showing th e av e rag e is also includ e d in th e illustration of FIG. 1 3. Oth e r choic e s in menu 
fi e ld 1312 would, for e xample, allow the human t e st e r to see re s ults in a line chart format 
as in FIG. 12 or in tabular format. 

Turning to FIG. 1 4, results in a tabular format are shown. Fi e ld 1312 indicat e s 
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that tho display format of "Log File" has been selected. This format corresponds to th e 

li s t shown in region 1 4 12. The list contains a column for th e namo of the methods in th e 

b o on b e ing test e d. In this e xample, th e data shown for each m e thod rev e als th e 

minimum, maximum and average execution time for that m e thod. 

As described above, t e st syst e m 110 measur es r e spons e tim e at various load 

conditions. The display e d data r e pr e sents r e spons e tim e s at a particular load 

condition. Thu s , to make th e list, the tester must specify the load condition for 

which data is to be displayed. To allow this selection to b e made, the page 

display e d in FIG. 11 contains a field 1410. In this fi e ld, a user can e nter the load 

condition for which data is to b e display e d. In this example, the human test e r has 

o nt o r o d a valu e of 500, indicating that 500 threads executing the test code were 

initiat e d in order to obtain th e display e d data. 

Having d e scribed th e structur e of test syst e m 1 10 and giving e xampl e s of its 

application, several important featur e s of the test syst e m 1 1 0 can be seen. One f e ature is 

that information about th e p e rformanc e of an application und e r tost can b e e asily 

obtain e d, with much of the data being deriv e d in an automated fashion. A software 

d e v e loper could us e th e t e st syst e m to find particular b e ans that arc lik e ly to b e 

performance bottlen e cks in an application. The developer could then r e write these beans 

or chang e th e ir d e ployment descriptors. For e xampl e , on e asp e ct of th e d e ploym e nt 

descriptor indicat e s the number of copi e s of th e b e an that ar e to be instantiated wi thin 

application under t e st 114. Th e dovelop e r could incr e as e the number of instantiations of 

a bean if that b e an is the bottleneck. 

The t e st syst e m d e scrib e d h e rein provid e s an easy and accurat e tool to test EJBs 
for scalability. It creates a us e r specifi e d numb e r of virtual users that call th e EJB 
while it is d e ployed on the applications server. The tool does thi s by inspecting 
tho EJB under t o st and automatically g e nerating a cli e nt t e st program, using either 
rules bas e d data or data suppli e d by an a human test e r, and th e n multithreading 
the cli e nt t e st program to driv e th e EJB und e r t e st. The r e sult is a seri e s of graphs 
r e porting on tho performanc e v e rsus th e number of users, which provide us e ful 
information in an easy to us e format. 

Anoth e r f e ature of th e inv e ntion is that th e tests ar c run without r e quiring changes 
in th e application under test or ev e n the installation of sp e cial t e st agents on the s e rv e r 
containing th e software under test. The g e nerat e d t e st code 516 ex e rcises the bean in the 
application und e r t e st using remote procedur e call s . 

Anoth e r f e ature of the d e scrib e d e mbodim e nt of t e st syst e m 1 10 is that it is 
scalable. To increas e tho number of tests that could simultan e ously bo run or th e siz e of 
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th e t e sts that could b e rim, mor e t e st e ngin e s oould b e add e d. Lik e wis e , mor e cod e 
g e n e rators could be add e d to support the simulation of a larger number of simul taneous 
us e rs. Th e specific numb e r of copi e s of each compon e nt is not important to the 
invention. Th e actual numb e r of e ach component in any given e mbodim e nt is lik e ly to 
vary from installation to installation. Th e mor e users an application is int e nded to 
support, the more test engines are likely to b e required. 

Anoth e r feature of th e describ e d e mbodiment is that testing is don e on th e 
simplest construct in th e application under test — the beans in th e illustrat e d e xample. 
Th e r e ar e two benefits to this approach. First, it allows tests to b e g e n e rat e d v e ry simply, 
with minimal human interv e ntion. S e cond, it allows a softwar e develop e r to focus in on 
th e point of th e software that n ee ds to b e chang e d or adjust e d in order to improve 


It should b e appr e ciat e d that displays of sp e cific kinds of infonnation hav e b ee n 
d e scribed. Various oth e r analyses might b e performed. It was describ e d that r e sponse 
tim e s and e rror rat e s as a function of load could be graph e d for display to a human tester 
for further analysis. One e nhanc e ment that might be made to test syst e m 1 1 0 is that the 
data analyz e r 218 could b e programm e d to perform furth e r analysis. It has b ee n 
r e cognized that, as th e load increas e s, th e r e is oft e n som e point at which th e performanc e 
of th e syst e m drastically chang e s. In som e instanc e s, th e tim e to compl e t e a transaction 
drastically incr e ases. A drastic incr e as e in transaction processing time indicates that the 
syst e m was not abl e to e ffectiv e ly handl e th e load. How e v e r, a d e cr e as e in proc e ssing 
time can also indicat e the load limit was r e ached. Sometim e s, a syst e m under test will 
r e spond with an error m e ssag e mor e quickly than it would tak e to g e n e rat e a correct 
respons e . Thus, if th e only param e ter being track e d is r e sponse time, a decreas e in 
processing tim e as a function of load can also signal that th e maximum load has be e n 
e xce e d e d. Of course, an increase in e rrors or error rate can also signal that the maximum 
load was e xce e d e d. Data analyz e r 21 8 could b e us e d to identify automatically a 
maximum load for a particular t e st cas e . By running multiple test cases, each t e st cas e 
focusing on a diff e r e nt b e an, t e st syst e m 1 10 could automatically determin e th e b e an that 
is the performance bottl e n e ck and could also assign a load rating to application und e r test 


Having d e scribed one e mbodim e nt, numerous alternative embodiments or 

variations might be mad e . For e xampl e , it was d e scrib e d that test syst e m 110 
automatically generate s t e st cod e to e x e rcis e b e ans that follow a patt e rn for database 
acc e ss. Th e s e b e ans are som e tim e s call e d "entity b e ans." In g e n e ral, th e r e will b e other 
b e ans in an application that p e rform computations on the data or that control the timing 
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of th e e x e cution of th e e ntity b e ans. Th e s e b e ans ar e sometimes call e d "s e ssion b e an s ." 
S e s s ion beans are less likely to follow prescribed programming patt e rns that mak e th e 
g e n e ration of t e st cod e for e ntity b e ans simpl e . As a r e sult, th e automatically g e nerat e d 
t e st cod e for s e ssion beans might not fully test those beans. In the describ e d 
e mbodim e nt, it is e xp e ct e d that th e human t e st e r supply t e st cod e to t e st s e ssion b e ans 
wh e r e th e automatically gen e rated te s ts are inadequate. 

On e possibl e modification to th e d e scrib e d e mbodim e nt is that th e compl e t e n e ss 
of t e sts for session beans might be increased. One way to increase the accuracy of tests 
for s e ssion b e ans would b e to captur e data about th e e x e cution of thos e b e ans during 
actual operation of the application under test 114. This data could allow an automat e d 
syst e m to d e t e rmine things lik e appropriat e data valu e s, which might th e n b e us e d to 
build a data table. Or, the captured data could allow th e automat e d s ystem to determine 
th e ord e r in which a s e ssion b e an acc e ss e s oth e r s e ssion b e ans or e ntity beans to cr e at e a 
r e alistic test. 

Also, as d e scrib e d, t e st cod e is g e n e rat e d to t e st a particular b e an, which is a 

simpl e construct or "component" of the application und e r t e st. Th e t e sting could focus 
on diff e r e nt constructs, such as sp e cific m e thods in a b e an. T e st cod e could b e g e n e rat e d 
to t e st specific methods within beans. Or, it was d e scrib e d that th e system records start 
and stop tim e of th e e x e cution of th e t e st cod e . The tim e s of oth e r ev e nts could b e 
record e d instead or in addition. — For e xampl e , start and stop tim e s of individual methods 
might b e r e cord e d, allowing p e rformanc e of individual m e thods to b e det e rmin e d. 

Alternatively, th e compl e xity of th e constructs b e ing t e st e d could b e increas e d. 

Multiple b e ans might b e test e d simultan e ously to d e t e rmin e int e ractions b e tw ee n b e ans. 
For e xampl e , multiple test cases might be ex e cuted at the same time, with one t e st cas e 
e x e rcising a sp e cifi e d instanc e s of on e b e an and a diff e r e nt t e st cas e e xercising a 
specifi e d number of instanc e s of a s e cond bean. 

As anoth e r e xampl e of a possibl e variation, it was d e scrib e d that a human t e st e r 

can insert code into a t e mplat e to do such thing s as put the application under test into a 
pr e dictabl e stat e . Lin e s of cod e might b e ins e rt e d dir e ctly, for exampl e by th e us e r 
simply typing th e lines of cod e . Or, th e t e st e r might ins e rt a "tag" into th e t e mplate. The 
tag would id e ntify a code s e gm e nt stored els e wh e r e within th e t e st syst e m. In this way, 
th e sam e code s e gm e nt could be includ e d at multipl e plac e s in th e t e mplate or in multiple 
t e mplates. 

As another e xample of possibl e variations, th e number of templates used to 

construct test code might b e vari e d. On e possibility is that e ach t e mplat e contain s all of 
th e steps n ee ded to initialize, run and t e rminat e a t e st. Thus, t e st code would be cr e at e d 
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by filling in a single t e mplat e . Alt e rnativ e ly, e ach t e mplat e might contain only th e st e ps 
n ee ded to p e rform on e function, such as initialization, testing or t e rmination. In this 
impl e m e ntation, test cod e would b e cr e at e d by stringing togeth e r multipl e t e mplat e s. 

Also, it was describ e d that in running a test that a numb e r of simultan e ous us e rs i s 

"synchroniz e d" — Simultan e ous us e rs ar e simulat e d by synchronizing copi e s of th e t e st 
cod e on differ e nt s ervers and on the same server. The term " s ynchronized" should not b e 
int e rpr e t e d in so limit e d a way as to imply that multipl e copi e s ar e e ach p e rforming 
e xactly th e same function at exactly the same tim e . Thus, when described herein that 
e x e cution is synchroniz e d, all that is r e quir e d is that e ach copy of th e cod e is making 
r e quest s of th e application und e r test during the window of time wh e n th e test is b e ing 
e x e cut e d. Som e copi e s of th e cod e will lik e ly start e x e cution soon e r or e nd sooner than 
the others. How e ver, as long as th e r e is ov e rlap in th e timing of e x e cution, th e test 
programs can b e said to b e synchroniz e d or running concurr e ntly. 

As a further variation, it was describ e d that th e test syst e m 1 1 0 provid e s outputs 

indicating th e p e rformance of an application und e r t e st as a function of load. Th e s e 
outputs in graphical or tabular form can b e us e d by an application developer to identify a 
numb e r of concurr e nt us e rs at which probl e ms with th e application ar e lik e ly to be 
e ncount e r e d. Pot e ntial probl e ms are manifested in various ways, such as by a sudden 
chang e in r e sponse tim e or error rat e as a function of load. T e st syst e m 110 could readily 
be programmed to automatically id e ntify patt e rns in th e output indicating th e s e problem 
points. 

Another us e ful modification would allow t e st system 1 10 to aid in identifying 

s e ttings for various param e t e rs in th e d e ploym e nt d e scriptor. As d e scrib e d abov e , the 
d e ployment descriptor for a b e an id e ntifies param e t e rs such as memory usage and a 
"pooling numb e r" indicating th e numb e r of instanc e s of a b e an that ar e cr e at e d at th e 
initialization of an application. Thes e and other settings in the deploym e nt descriptor 
might hav e an impact on the p e rformanc e tim e and maximum load that an application 
could handl e . One us e of th e t e st s ystem described above is that it allows a test cas e to be 
r e peat e d for diff e r e nt s e ttings in th e d e ploym e nt d e scriptor. A human t e ster can analyz e 
changes in p e rformanc e for diff e r e nt settings in th e d e ploym e nt descriptor. However, 
t e st syst e m 110 could b e programm e d to automatically e dit th e d e ploym e nt d e scriptor of 
a bean by changing param e t e rs aff e cting pooling or m e mory usage. T e st system f 10 
could then automatically gath e r and pr e s e nt data showing th e impact of a d e ployment 
d e scriptor on p e rformance of an application. 

Even higher l e v e ls of automation could b e achi e v e d by t e st system 110. For 

e xample, test system 1 1 0 might t e st th e b e ans in an application and analyz e the r e sults of 
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t e sting e ach b e an. T e st syst e m 110 might id e ntify th e b e an or b e ans that r e fl e ct 
performanc e bottlenecks (i.e. that exhibited unacceptable response timos for the lowe s t 
numb e rs of simultaneous users). Th e n, tost syst e m 110 could run t e sts on thos e b e ans to 
find settings in the d e ployment descriptors that would balanc e the performance of th e 
b e ans in th e application (i. e . to adaptiv e ly adjust the s e ttings in th e d e ployment 
d e scriptors so that the bottleneck beans performed no wors e than other b o ans.) 

It should also b e appr e ciat e d that comput e r t e chnology is rapidly e volving and 

improved or enhanced versions of the hardware and software components making up the 
application und e r t e st and th e t e st syst e m ar e likely to b e com e available. It should also 
be appreciat e d that the d e scription of one device in a class is intended to be illustrati ve 
rath e r than limiting and that oth e r d e vic e s within th e sam e class might b e substitut e d with 
ordinary skill in the art. For exampl e , the application under test was described in the 
cont e xt of a conv e ntional application acc e ss e d through a cli e nt on a PC using a web 
browser as a graphical u se r interfac e . It should b e appreciated, though, that if the cli e nts 
are intend e d to b e hous e hold applianc e s with micro controll e rs, a diff e r e nt int e rfac e 
might be readily substitut e d for th e graphical user interface, 

Also, it was d e scribed that FIG. 1 1 shows summary data of a t e st aft e r e x e cution 

is complet e d. It will b e appreciat e d, though, that data on a test case execution might be 
displayed to a human t e st e r whil e a t e st is in proc e ss. For e xampl e , th e summary scr ee n 
might contain a field that shows the percentage of the t e st case that is complet e d. This 
valu e would updat e as th e t e st runs. Likewis e , the values for av e rag e and maximum 
r e sponse tim e could b e updat e d as th e data is gathered. 

Also, it was d e scrib e d that th e obj e cts b e ing test e d ar e EJBs, which ar e writt e n in 

the Java language. The same techniques ar e e qually applicable to applications having 
compon e nts impl e m e nt e d in oth e r languages. For e xampl e , applications written 
according to the COM standard might be writt e n in Visual Basic and applications writt e n 
for th e CORBA standard might b e written in C++. 

Regardl e ss of the specific languag e us e d, th e se standards ar e intend e d to allow 
s e parat e ly d e v e lop e d compon e nts to op e rat e tog e ther. Thus, e ach must provid e a 
m e chanism for other application s , such as t e st system 1 10, to d e t e rmine how to acc e ss the 
m e thods and prop e rti e s of th e ir components. How e v e r, th e r e could b e diff e r e nc e s in th e 
specific commands used to acc e ss compon e nts. 

In on e e mbodim e nt, cod e g e n e rator 212 is impl e m e nt e d in a way that will mak e it 
e asy to modify for g e n e rating test code for applications written in a different languag e . 
Sp e cifically, cod e gen e rator 212 stor e s interm e diat e r e sults as a symbol tabl e that is 
ind e p e ndent of the sp e cific languag e used to program th e application und e r t e st. The 
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symbol tabl e lists m e thods and prop e rti e s for the component b e ing t e st e d. Wh e n to 
acc e ss th e s e methods and what data to use for a particular test and what kinds of data to 
r e cord can b e d e t e rmined from th e information in th e symbol tabl e and input from th e 
us e r. Thus, much of the functioning of cod e g e nerator 212 is independent of the specific 
languag e us e d to implem e nt th e application und e r t e st. 

In this way, the language sp e cific asp e cts of cod e generator 212 ar e easily 
s e gr e gat e d and r e pr e s e nt a r e lativ e ly small part of the cod e g e n e rator 212. In particular, 
language specific information is needed to access the application under test to derive the 
information for th e symbol tabl e . Languag e sp e cific information is also n ee d e d to format 
the generat e d client test code. But, it i s int e nded that these parts of code g e nerator 212 
could b e r e plac e d to allow t e st syst e m 1 10 to t e st applications writt e n in oth e r languag e s. 
Also, it is possible that test syst e m 1 10 will contain multiple versions of the language 
sp e cific parts and th e us e r could sp e cify as an input the languag e of th e application und e r 

Th e r e for e , th e inv e ntion should b e limit e d only by th e spirit and scop e of th e 

app e nded claims. 

METHOD AND SYSTEM FOR TESTING SOFTWARE COMPONENTS 


This invention relates generally to computer software applications and more 
specifically to testing computer software applications. 

Distributed computing has been used for many years. Distributed computing is 

very prevalently used in "enterprise- wide" applications. An enterprise- wide application 
is an application that allows a large group of people to work together on a common task. 
Usually, an enterprise-wide application performs functions that are essential to a 
company's business. For example, in a bank, people at every bank branch must be able 
to access a database of accounts for every bank customer. Likewise, at an insurance 
company, people all over the company must be able to access a database containing 
information about every policyholder. The software that performs these functions is 
generally known as enterprise-wide applications. 

As available hardware and software has evolved, the architecture of enterprise 
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wide applications has changed. An architecture which is currently popular is called the 
N-Tier enterprise model The most prevalent N-tier enterprise model is a three tier 
model The three tiers are the front end, the middleware and the back end. The back end 
is the database. The front end is sometimes referred to as a "client" or a Graphical User 
Interface (GUP. The middleware is the software that manages interactions with the 
database and captures the "business logic," Business logic tells the system how to 
validate, process and report on the data in a fashion that is useful for the people in the 
enterprise. 

The middleware resides on a computer called a server. The database might be on 
the same computer or a different computer. The "client" is usually on an individual's 
personal computer. All of the computers are connected together through a network. 
Because many people use the enterprise wide application, such systems are set up to 
allow simultaneous users and there would be many clients connected to a single server. 
Often, many clients will be connected to the server simultaneously. 

Those familiar with Internet commerce will recognize that the N-tier ed model 
also describes many Internet web sites that sell goods or services. For example, a web 
site that auctions cars is likely to fit the N-tiered model In such an application, databases 
are provided to track buyers, sellers and objects being auctioned. Also, a database must 
be provided to track the bids as thev are entered. The middleware provides the access to 
these databases and encapsulates the business logic around such transactions a s when to 
accept a bid, when to declare an item sold, etc. In the world of distributed computing, it 
makes no difference whether the "clients" using the application are em ployees of a single 
company or many Internet users throughout the world. Herein, examples of applications 
under test will be given, but they are not intended to imply limitations on the use of the 
invention. The inventions described herein could be used by developers of e nterprise- 
wide applications or web based applications. 

One advancement in the N-tiered model is that the middleware is very likely to be 
componentized and is very likely to be written to a component standard so that it will 
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easily integrate with software at other tiers. Enterprise JavaBeans fEJB technology based 
software components) by Sun Microsystems, COM, DCOM and COM+ by Microsoft 
Corporation and CORBA by IBM are examples of component specification standards 
that are commercially available. Herein, EJB technology based software component is 
used as an example of a component standard used to implement middleware in an N- 
tiered model , but it should be appreciated that and the concepts described herein could be 
used with other component standards. 

EJB technology based software components are written in the JAVA language, 
which is intended to be "platform independent." Platform independent means that an 
application is intended to perform the same regardless of the hardware and operating 
system on which it is operating. Platform independence is achieved through the use of a 
"container." A container is software that is designed for a specific platform. It provides 
a standardized environment that ensures the application written in the platform 
independent language operates correctly. The container is usually commercially 
available software and the application developer will buy the container rather create it. 

V 

Componentized software is software that is designed to allow different pieces of 
the application, or "objects", to be created separately but still to have the objects work 
together. For this to happen, the objects must have standard interfaces that can be 
understood and accessed by other objects. Some parts of these interfaces are enforced by 
the software language. If the interfaces are not used, the software objects will not be able 
to work with other objects. Other practices are imposed by convention. Usually, one 
company has "control" over the language and specifies programming practices that 
should be followed by anyone writing platform independent software in that language. 
Because these programming practices are known to everyone, the companies that create 
the containers can rely on them when creating the container. As a result, if these 
practices are not followed, the container might not operate properly. Thus, there is an 
indirect mechanism for enforcing these practices. 

Typically, applications have been tested in one of two ways. The objects are 
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tested as they are written. Each is tested to ensure that it performs the intended function. 
When the objects are assembled into a completed application, the entire application is 
then usually tested. Heretofore, application testing has generally been done by applying 
test inputs at the client end and observing the response of the application. There are 
several shortcomings with this process. One is that it is relatively labor intensive, 
particularly to develop a load or scalability test. There has been no easy way to create 
the test program, instantiate it with test data, execute the test and aggregate the results. 

Some tools, called "profilers," have been available. However, these tools track 
things such as disk usage, memory usage or thread usage of the application under test. 
They do not provide data about performance of the application based on load. 

Other tools are available to automate the execution of tests on applications. For 
example, RSW Software, Inc. of Waltham, MA, provides a product called g-Load. This 
tool simulates load on an application under test and provides information about the 
performance of the application. However, this tool does not provide information about 
the components in an application. We have recognized that a software developer would 
find such information very useful. 

Automatic test generation tools, such as TestMaster available from Teradyne 
Software and System Test of Nashua, NH, are also available. Tools of this type provide a 
means to reduce the manual effort of generating a test. TestMaster works from a state 
model of the application under test. Such an application is very useful for generating 
functional tests during the development of an application. Once the model of the 
application is specified, TestMaster can be instructed to generate a suite of tests that can 
be tailored for a particular task - such as to fully exercise some portion of the application 
that has been changed. Model based testing is particularly useful for functional testing of 
large applications, but is not fully automatic because it requires the creation of a state 
model of the application being tested. 

We have recognized that a second shortcoming of testing enterprise wide 
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applications is the critical performance criteria to measure often relates to how the 
application behaves as the number of simultaneous users increases. There are examples 
of websites crashing or operating so slow as to frustrate an ordinary user when too many 
users log on simultaneously. In the past load has been simulated informally, such as by 
having several people try to use the application at the same time. Some tools exist to 
provide a load on an application for testing, such as e-Load available from RSW of 
Waltham, MA. 
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However, it has generally not been until the application is deployed into its intended 
operating environment that the performance of the application under load is known. 
Thus, the biggest problem facing an application developer might not be testing to see 
whether each object performs as designed or even whether the objects work together as a 
system. Heretofore there has been no available tool that will help an application 
developer ascertain how many simultaneous users a middleware application can 
accommodate given a specified transaction response time or identify which object in the 
application, given real world load conditions, is causing the bottleneck.SUMMARY OF 
THE INVENTION 

With the foregoing background in mind it is an object of the invention to provide 

testing tools to facilitate load based testing of N-tiered applications. 

It is also an object to provide automatic testing. 

The foregoing and other objects are achieved by a test system that generates test 

code for components of an application under test using interface information about the 
components. 

In the preferred embodiment, the generated test code simulates use of a particular 

software object within an application by a plurality of simultaneous users. The number 
of simultaneous users simulated is varied. 

In a presently preferred embodiments, the test code is generated from interface 
information for the object under test, which identifies methods of the object. Ranges and 
formats for data values for the inputs and outputs of the methods are determined from 
properties specified in the interface information. Specific data values are generated from 
user supplied tables or using algorithms to pick allowable values within the range. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The invention will be better understood by reference to the following more 

detailed description and accompanying drawings in which 

FIG. 1 is an illustration of an application under test by the test system of the 
invention; 

FIG. 2 is an illustration showing the test system of the invention in greater detail; 
FIG. 3 is an illustration showing the coordinator of FIG. 2 in greater detail; 
FIG. 4 is a flow chart illustrating the process of coordinating execution of load 
tests; 

FIG. 5 is an illustration showing the code generator of FIG, 2 in greater detail; 
FIG. 6 illustrates a user interface of test system 110 during a setup phase; 
FIG. 7 illustrates a user interface of test system 1 10 during the specification of a 
test case; 

FIG. 8 illustrates a user interface of test system 110 during a different action as 

part of the specification of a test case; 
FIG. 9 illustrates a user interface of test system 110 during a different action as 

part of the specification of a test case; 
FIG. 10 illustrates a user interface of test system 110 during a different action as 

part of the specification of a test case; 
FIG. 1 1 illustrates a user interface of test system 110 during a phase for reviewing 

the results of a test case in tabular format; 
FIG. 12 illustrates a user interface of test system 110 during a phase for reviewing 

the results of a test case in graphical format; 
FIG. 13 illustrates a user interface of test system 110 during a phase for reviewing 

the results of a test case in an alternative graphical format; and 
FIG. 14 illustrates a user interface of test system 1 10 during a phase for reviewing 

the results of a test case in a tabular format. 
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DESCRIPTION OF THE PREFERRED EMBODIMENT 

FIG, 1 illustrates a test system IIP according to the present invention. The 

system is testing application under test 114. Here application under test 114 is an 
application in the N-tiered model. More specifically, it is a three tiered database 
application. Application under test 1 14 could represent a database for a bank or an 
insurance company or it could represent an Internet application. The specific function of 
application under test 1 14 are not important to the invention. 

Also, the specific hardware on which test system HQ and the application under 

test 114 reside is not important to the invention. It is sufficient if there is some 
connection between the two. In the illustration, that connection is provided by network 
122. In the illustrated embodiment it is contemplated that network 122 is part of a LAN 
operated by the owner of application under test 114, such as an Ethernet network. In this 
scenario, test system 110 is installed on a server within the LAN. However, many other 
implementations are possible. For example, network 122 could be a WAN owned by the 
owner of application under test 1 14. 

A further variation is that network 122 could the Internet. In that scenario, test 
system 110 could be located in a server owned by a testing company. 

A further possibility is that test system and application under test could be located 
in computers owned by a testing company. Many applications are written using platform 
independent technology such that the application will perform the same on many 
different platforms. Platform independent technology is intended to make it easy to run 
an application on any platform. Therefore, the application under test could be sent to a 
testing company, owning the hardware to implement test system 110, such as by 
uploading over the Internet. Thereafter, the application under test could be tested as 
described herein while running on a platform provided by the testing company with the 
results of the test being downloaded over the Internet. 
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Still other variations are possible. Test system 110 and application under test 1 14 
could physically be implemented in the same computer. However, that implementation is 
not presently preferred because a single computer would have to be very large or would 
be Limited in the size of applications that could be tested. The presently preferred 
embodiment uses several computers to implement test system 1 10. 

Application under test 1 14 is a software application as known in the art. It 

includes middleware 116 that encapsulates some business logic. A user accesses the 
application through a client device. Many types of client devices are possible, with the 
list growing as networks become more prevalent. Personal computers, telephone systems 
and even household appliances with micro- controllers could be the client device. For 
simplicity, the client device is illustrated herein as a personal computer (PC) 120, though 
the specific type of client device is not important to the invention. PC 120 is connected 
to network 122 and can therefore access application under test 114. In use, it is 
contemplated that there would be multiple users connected to application under test 114, 
but only one user is shown for simplicity. The number of users simultaneously accessing 
application under test 1 14 is one indication of the "load" on the application. 

Access to the application under test is, in the illustrated embodiment, through 

Graphical User Interface (GUI) 124 of the type known in the art. Software to manage 
interactions between multiple users and an application is known. Such software is 
sometimes called a web server. Web servers operate in conjunction with a browser, 
which is software commonly found on most PC's. 

The web server and browser exchange information in a standard format known as 

HTML. An HTML file contains tags, with specific information associated with each tag. 
The tag signals to the browser a type associated with the information, which allows the 
browser to display the information in an appropriate format. For example, the tag might 
signal whether the information is a title for the page or whether the information is a link 
to another web page. The browser creates a screen display in a particular window 
running on PC 120 based on one or more HTML pages sent by the web server. 
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When a iiser inputs commands or data into the window of the browser, the 

browser uses the information on the HTML page to format this information and send it to 
the web server. In this way, the web server knows how to process the commands and 
data that comes from the user. 

GUI 124 passes the information and commands it receives on to middleware 1 16. 

In the example of FIG, 1, middleware 116 is depicted as a middleware application 
created with EJB technology based software components. Containers 130 are, in. a 
preferred embodiment, commercially available containers. Within a container, are 
numerous enterprise Java beans 132. Each Java bean can more generally be thought of as 
a component. GUI 124, based on the information from user PC 120. passes the 
information to the appropriate EJB technology based software component 132. Outputs 
from application under test 1 14 are provided back through GUI 124 to PC 120 for display 
to a user. 

EJB technology based software component's 132, in the illustrated example. 

collectively implement a database application. EJB technology based software 
component's 132 manage interactions with and process data from databases 126 and 128. 
They will perform such database functions as setting values in a particular record or 
getting values from a particular record. Other functions are creating rows in the database 
and finding rows in the database. EJB technology based software component's that 
access the database are often referred to as "entity beans." 

Other types of EJB technology based software component's perform computation 

or control functions. These are called "session beans." Session beans perform such 
functions as validating data entries or reporting to a user that an entry is erroneous. 
Session beans generally call entity beans to perform database access. 

It will be appreciated that, while it is generally preferable to segregate 
programming of the application in such a way that each type of database transaction is 
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controlled by a single bean that performs only that function, some entity beans will 
perform functions not strictly tied to database access. Likewise, some session beans will 
perform database access functions without calling an entity bean. Thus, while different 
testing techniques will be described herein for testing session beans and entity beans, it is 
possible that some EJB technology based software component's will have attributes of 
both entity and session beans. Consequently, a full test of any bean might employ 
techniques of testing entity beans and testing session beans. 

Test system 1 10 is able to access the EJB technology based software component's 
132 of application under test 114 over network 122, In this way, each bean can be 
exercised for testing. In the preferred embodiment, the tests are predominately directed 
at determining the response time of the beans - or more generally determining the 
response time of components or objects used to create the application under test. 
Knowing the response time of a bean can allow conclusions about the performance of an 
application. The details of test system 1 10 are described below. 

In the illustrated embodiment, test system 1 10 is software installed on one or 
more servers. It is conceptually much like application under test 114. In a preferred 
embodiment, test system 1 1 0 is a J AV A application. Like application under test 1 14, test 
system 1 10 is controlled through a graphical user interface 1 50. GUI 1 50 might be a web 
server as known in the art. One or more application developers or test engineers might 
access test system over the network 122. In FIG. 1, PCs 152 and 154 are PC's used by 
testers who will control the test process. 

Like application under test 114, multiple individuals might use test system 1 10 
simultaneously. For example, multiple testers might be testing a single application. Each 
tester might be focused on testing different aspects of the application. Alternatively, each 
tester might be testing a different application. Numerous applications might be installed 
on computers on network 122. Test system HQ might be testing different applications at 
the same time. 
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Turning now to FIG, 2, details of test system 110 arc shown. Test system 110 
performs several functions. One function is the generation of test code. A second 
function is to execute the test code to exercise one or more EJB technology based 
software component's in the application under test. Another function is to record 
and analyze the results of executing the test code. These functions are performed 
by software running on one or more computers connected to network 122. The 
software is written using a commercially available language to perform the 
functions described herein. 

FIG. 2 shows that test system 110 has a distributed architecture. Software 
components are installed on several different computers. Multiple computers are used 
both to provide capability for multiple users, to a allow a user to perform multiple tasks 
and also to run very large tests. The specific number of computers and the distribution of 
software components of the test system on those computers is not important to the 
invention. 

Coordinator 210 is a software application that interfaces with GUI 1 50. The main 
purpose of coordinator 210 is to route user requests to an appropriate server in a fashion 
that is transparent to a user. Turning to FIG. 3, coordinator 210 is shown in greater 
detail. It should be appreciated, though, that FIG. 3 shows the conceptual structure of 
coordinator 210. Coordinator 210 might not be a single, separately identified piece of 
software. It might, for example, be implemented as coordination software within the 
various other components of test system 110. Also, it should be realized that a web 
server used to implement GUI 150 also provides coordination functions, such as queuing 
multiple requests from an individual or coordinating multiple users. 

Coordinator 210 contains distribution unit 3 12. Distribution unit 312 is preferably 
a software program running on a server. As user requests are received from GUI 150. 
they are received by distribution unit 312. As the requests are received, distribution unit 
312 determines the type of resource needed to process the request. For example, a 
request to generate code must be sent to a server that is running a code generator. 
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Coordinator 210 includes several queues to hold the pending requests. Each 
queue is implemented in the memory of the server implementing coordinator 210. In 
FIG. 3. queues 318A...318C are illustrated. Each queue 318A...318C corresponds to a 
particular type of resource. For example, queue 318A could contain code generator 
requests, queue 318B could contain test engine requests and queue 318C could contain 
data analysis requests. Distribution unit sends each request to one of the queues 
3 1 8 A. . .3 1 8C, based on the type of resources needed to process the request. 

Associated with each queue 318A...318C is queue manager 320A...320C. Each 
queue manager is preferably implemented as software running on the server 
implementing coordinator 210 or the server implementing the relevant piece of 
coordinator 210. Each queue manager maintains a list of servers within test system 110 
that can respond to the requests in its associated queue. A queue manager sends the 
request at the top of the queue to a server that is equipped to handle the request. The 
connection between the queue manager and the servers equipped to handle the requests is 
over network 122. If there are other servers available and still more requests in the 
queue, the queue manager will send the next request in the queue to an available server. 
When there are no available servers, each queue manager waits for one of the servers to 
complete the processing of its assigned request. 

As the requests are processed, the servers, such as the code generators and the test 
engines report back to the queue managers. In response, the queue managers send 
another request from the queue and also provide the results back to the distribution unit 
312. Distribution unit 312 can then reply back to the user that issued the request, 
indicating that the request was completed and either giving the results or giving the 
location of the results. For example, after test code is generated, the user might receive 
an indication of where the test code is stored. After a test is executed, the user might 
receive a report of the average execution time for the test or the location of a file storing 
each measurement made during the test. 
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It will be appreciated by one of skill in the art that software systems that process 
user commands, including commands from multiple users, are well known. Such 
systems must have an interface for receiving commands from a user, processing those 
commands and presenting results to the user. Such interfaces also allow those results to 
be used by the user for implementing further commands. Such an interface is employed 
here as well and is depicted generally as GUI 150, For example, GUI 150 will allow a 
user to enter a command that indicates code should be generated to test a particular 
a pplication. Once the code is generated, GUI 150 allows the user to specify that a test 
should be run using that test code. 

It is possible that some requests will require the coordination of multiple hardware 
elements. As will be described hereafter, one of the functions of the test engines is to 
a pply a load that simulates multiple users. In some instances, one computer can simulate 
multiple users by running multiple client threads. However, there is a limit to the number 
of client threads that can run on a server. 

Multiple threading the client test program has the advantage of being able to 

generate very large loads (tens of thousands to hundred of thousands virtual users). As 
such, multiple threading the client test program accurately emulates multiple users as far 
as the software component (EJB technology based software component) deployed on the 
a pplication server is concerned. Only one socket is established between the host process 
and the application server and all the threads communicate with the software object 
through the socket. 

A Java Virtual Machines (JVM) is a software implementation of a processor 
designed to run Java code. Each test engine can run one or more JVMs, with each JVM 
having a respective socket to the application server. The JVM software implementation 
of a processor runs within a process. The JVM software implementation of a processor 
acts as an interface between compiled Java binary code (bytecode) and the 
microprocessor. Since each JVM software implementation of a processor has a 
respective socket to the application server, the use of JVM software implementation of a 
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processors very accurately emulates a large user load on the application server and on the 
software components. Each test engine can run multiple JVM software implementation 
of a processors, and each JVM software implementation of a processor can run multiple 
threads. The test engines can run any mixture of multiple threads and multiple JVM 
software implementation of a processors. The test engines can also run a multithreaded 
load from multiple processes, applicable to testing EJB technology based software 
components and COM+ (Component Object Modules). 

For distributed testing, the entire class file path specified by the user is provided 
at the remote machines. The entire class file path is compressed, also known as put into a 
jar, and the compressed file or jar is sent to the remote machine. In such a manner testing 
software components with distributed execution is greatly simplified. 

FIG. 4 shows the process by which queue manager 320B coordinates the actions 
of test engines located on separate servers. At step 410, queue manager 320B waits for 
the required number of test engines to become available. Once the test engines are 
available, at step 412 queue manager 320B sends commands to each test engine that will 
be involved in the test to download the test code from the appropriate one of the code 
generators 212A and 21 2 B. 

Queue manager 320B then begins the process of synchronizing the test engines 
located on different servers. Various methods could be used to synchronize the 
servers. For example, if very great accuracy is required, each server could be 
equipped with radio receivers that receive satellite transmissions that are intended 
to act as time reference signals, such as in a GPS system. Calibration processes 
could alternatively be used to determine the amount of time it takes for a 
command to reach and be processed by each server. Commands could then be 
sent to each server at times offset to make up for the time differences. In the 
preferred embodiment, a simple process is used. At step 414, queue manager 
sends a message to each server to be acting as a test engine. The message asks 
that server to report the time as kept by its own internal circuitry. 
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The multiple JVM software implementation of a processors can be synchronized 
by loading an agent on each system the JVM software implementation of a processors 
will run, then the server sends a message out to the agents and requests a time check. The 
server then sends the client code to the JVM software implementation of a processors on 
the remote machines along with the time at which the JVM software implementation of a 
processors should start executing the client code. In this manner highly accurate and 
repeatable load generation is achieved. Additionally, the user can specify a time offset 
for each of the machines and JVM software implementation of a processors. For 
example, a first machine would be started; a second machine stalled ten seconds later, 
etc. While this process is independent of the absolute time on each system, the machines 
could be directed to start at a predefined time based on the system time. 

Additional synchronization techniques include where the server sends the client 
code to the host, the server does a time check, then the server sends the start time to the 
host. Alternately the server could send the client code and start time to the hot. Yet 
another synchronization technique includes having a service (e.g. JIND running on the 
host computer. The server sends a message to synchronize in "X" number of ticks. The 
services wait for "X" number of ticks then executes the client code. Further, the JVM 
software implementation of a processors could run in an unsynchronized mode, wherein 
the JVM software implementation of a processors start running the client code 
independently of other J VM software implementation of a processors 

Upon receiving the internal time as kept by each of the servers, at step 416 queue 
manager 320B adds the same offset to each local time. At step 418, queue manager 418 
sends the offset times back to the servers. The offset local times become the local starting 
time of the test for each server. Each server is instructed to start the test when its local 
time matches the offset local time. In this way, all the servers start the test 
simultaneously. 

Queue manager 320B waits for the execution of the test code at block 420. Some 
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test cases will entail multiple tests. At step 422, a check is made of whether the particular 
test case being executed requires more tests. For example, as will be described In greater 
detail below, one kind of test case that test system 110 will run is a load test. During a 
load test, multiple test engines, each executing multiple client threads, execute to 
simulate multiple users accessing application under test 114. An operating parameter of 
application under test 1 14 is then measured. In the preferred embodiment, the number of 
simultaneous users being simulated can be varied and the operating parameter measured 
again. Taking data at multiple load conditions allows test system 110 to determine the 
affect of load on application under test 114. Measurements of these types require that a 
ful l test case include multiple repetitions of the same test wi th different load conditions. 

If there are more conditions under which the test should be run, execution 
proceeds to step 424. At step 424, the number of client threads is increased as needed for 
the new test case. Execution then loops back to step 410. At step 410. queue manager 
320B waits for the required number of test engines to be available. When the hard ware is 
available, the test is performed through steps 412, 414. 416, 418 and 420 in the same 
manner as for the first test condition. At step 422, a check is for whether the request 
entails multiple test conditions. If there are no further test conditions, the request from 
queue 318B is considered complete. If there are farther test conditions that need to be 
am to complete the request, the process is again repeated. 

For test system 1 10 to operate, it is necessary that there be test code. A user could 
provide test code. Or, test code could be provided by automatic code generation systems, 
such as TESTMASTER sold by Teradyne Software and System Test of Nashua, NH. 
However, FIG. 2 illustrates that code generators 212A and 212B are used in the preferred 
embodiment to create the code. Turning to FIG. 5, greater details of a code generator 212 
are shown. 

Code generator 212 contains several scripts 512. Each script is a sequence of 
steps that code generator 212 must perform to create code that performs a certain type of 
test. The scripts can be prepared in any convenient programming language. For each 
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type of test that test system 1 1 0 will perform, a script is provided. User input on the type 
of test that is desired specifies which script 512 is to be used for generating code at any 
given time. 

The selected script 512 assembles test code 516, The information needed to 
assemble test code 516 comes from several sources. One source of information is the test 
templates 514. There are some steps that are needed in almost any kind of test. For 
example, the object being tested must be deployed and some initialization sequence is 
required. If the tests are timed, there must be code that stalls the test at a specified start 
time and an ending time of the test must be recorded. Also, there must be code that 
causes the required data to be logged during the test. After the test, there might also be 
some termination steps that are required. For example, where the initialization started 
with a request for a reference to a particular EJB technology based software component, 
the test code will likely terminate with that reference being released. The test code to 
cause these steps to be performed is captured in the set of templates 514. 

In addition, there might be different templates to ensure that the test code 516 
appropriately reflects inputs provided by the user. For example, different containers 
might require different command formats to achieve the same result. One way these 
different formats can be reflected in the test code 516 is by having different templates for 
each container. Alternatively, a user might be able to specify the type of information that 
is to be recorded during a test. In that instance, a data logging preference might be 
implemented by having a set of templates that differ in the command lines that cause data 
to be recorded during a test. 

The templates are written so that certain spaces can be filled in to customize the 
code for the specific object to be tested. In the preferred embodiment, code generator 
212 generates code to test a specific EJB technology based software component in an 
application under test. One piece of information that will need to be filled in for many 
templates is a description of the EJB technology based software component being tested. 
Another piece of information that might be included is user code to put the application 
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under test in the appropriate state for a test. For example, in testing a component of an 
application that manages a database of account information for a bank, it might be 
necessary to have a specific account created in the database to use for test purposes or it 
might otherwise be necessary to initialize an application before testing it. The code 
needed to cause these events might be unique to the application and will therefore be best 
inserted into the test code by the tester testing the application. In the illustrated 
embodiment, this code is inserted into the template and is then carried through to the final 
test code. 

The template might also contain spaces for a human tester to fill in other 
information, such as specific data sets to use for a test. However, in the presently 
preferred embodiment, data sets are provided by the human user in the form of a data 
table. 

DataTables are used to apply large datasets to the application under test. In order 
to properly load test software components, it is necessary to specify large datasets. The 
software component is inspected and a template is provided. This template may be in a 
.CSV (comma separated value) format although other formats could also be used. In the 
datatable the columns are used for parameters and the rows representing each user. The 
software cycles through each row of data as each user. If there are fewer rows than 
virtual clients, the software will go to the end of the data set then start over beginning 
with the first row. 

Code generator 212 could generate functional tests. Functional tests are those 
tests that are directed at determining whether the bean correctly performs its required 
functions. In a functional test, the software under test is exercised with many test cases to 
ensure that it operates correctly in every state. Data tables indicating expected outputs 
for various inputs are used to create functional test software. However, in the presently 
preferred embodiment, code generator 212 primarily generates test code that performs 
load tests. In a load test, it is not necessary to stimulate the software under test to 
exercise every possible function and combination of functions the software is intended to 
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perform. Rather, it is usually sufficient to provide one test condition. The objective of 
the load test is to measure how operation of the software degrades as the number of 
simultaneous users of the application increases. 

In the preferred embodiment, test system 110 contains scripts 512 to implement 
various types of load tests. One type of load test determines response time of an EJB 
technology based software component. This allows the test system to vary the load on 
the EJB technology based software component and determine degradation of response 
time in response to increased load. Another type of load test is a regression type load 
test. In a regression type test, the script runs operations to determine whether the EJB 
technology based software component responds the same way as it did to some baseline 
stimulus. In general, the response to the baseline stimulus represents the correct 
operation of the EJB technology based software component. Having a regression type 
test allows the test system 1 10 to increase the load on a bean and determine the error rate 
as a function of load. 

To generate test code 516 for these types of load tests, the script 512 must create 
test code that is specific to the bean under test. The user provides information on which 
bean to test through GUI 1 50. In the preferred embodiment, this information is provided 
by the human tester providing the name of the file within the application under test that 
contains the "deployment descriptor'- for the specific bean under test. This information 
specifies where in the network to find the bean under test. Script 512 uses this 
information to ascertain what test code must be generated to test the bean. 

Script 512 can generate code by using the attributes of the platform independent 
language in which the bean is written. For the example of Sun JAVA language being 
used here, each bean has an application program interface called a "reflection." More 
particularly, each bean has a "home" interface and a "remote" interface. The "home" 
interface reveals information about the methods for creating or finding a remote interface 
in the bean. The remote interface reveals how this code can be accessed from client 
software. Of particular interest in the preferred embodiment, the home and remote 


interfaces provide the information needed to create a test program to access the bean. 

AutoGen creates tinier folders for each of the methods in a test. The methods are 

organized by topic (for example all Getter Methods are put into the Get Method folder). 
The results are displayed organized in the same folder layout that AutoGen and the user 
defined in the client program. This approach has the advantage that it is flexible and 
totally customizable by the user. Each parent folder calculates the average of the 
response time of each item within the folder and this works hierarchically. The parent 
folder may also perform other calculations (such as total, etc.) on the items within each 
folder. 

Using the reflection, any program can determine what are known as the 
' 'properties" and "methods" of a bean. The properties of a bean describe the data types 
and attributes for a variable used in the bean. Every variable used in the bean must have 
a property associated with it. In this way, script 512 can automatically determine what 
methods need to be exercised to test a bean and die variables that need to be generated in 
order to provide stimulus to the methods. The variables that will be by the methods as 
they are tested can also be determined. In the preferred embodiment, this information is 
stored in symbol table 515. 

Symbol table 515 is a file in any convenient file format. Many formats for storing 
tabular data are known, such as .xml format. Once the information on the methods and 
properties are captured in a table, script 512 can use this information to create test code 
that exercises the methods and properties of the particular component under test. In 
particular, script 512 can automatically create a variable of the correct data type and 
assign it a value consistent with that type for any variable used in the bean. 

FIG. 5 shows a data generator 518. Data generator 518 uses the information 
derived from the reflection interface to generate values for variables used during testing 
of a bean. There are many ways that appropriate values could be generated for each 
variable used in the test of a particular bean. However, in the commercial embodiment of 



the present invention, the user is given a choice of three different algorithms that data 
generator 518 will use to generate data values. The user can specify "maximum," 
"minimum" or "random." If the maximum choice is specified, data generator 518 
analyzes the property description obtained through the reflection interface and determines 
the maximum permissible value. If the user specifies "minimum" then data generator 
518 generates the smallest value possible. If the user specifies random, data generator 
518 selects a value at random between the maximum and the minimum. 

In many instances where a load test is desired, the exact value of a particular 
variable is not important. For example, when testing whether a bean can properly store 
and retrieve a value from a database, it usually does not matter what value is stored and 
retrieved. It only matters that the value that is read from the database is the same one that 
was stored. Or. when timing the operation of a particular bean, it will often not matter 
what values are input to the method. In these scenarios, data generator 518 can 
automatically generate the values for variables used in the test code. 

In cases where the specific values of the variables used in a test are important, 
code generator 212 provides the user with another option. Rather than derive values of 
variables from data generator 518, script 512 can be instructed to derive data values from 
a user provided data table 520. A user might, for example, want to provide a data table 
even for a load test when the execution time of a particular function would depend on the 
value of the input data. 

A data table is implemented simply as a file on one of the computers on network 
122. The entries in the table, specifying values for particular variables to use as inputs 
and outputs to particular methods, are separated by delimiters in the file. A standard 
format for such a table is "comma separated values" or CSV. In a preferred 
embodiment, test system 110 includes a file editor - of the type using conventional 
technology - for creating and editing such a file. In addition, test system 110 would 
likely include the ability to import a file - again using known techniques - that has the 
required format. 
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The methods of a bean describe the functions that bean can perform. Part of the 
description of the method is the properties of the variables that are inputs or outputs to the 
method. A second part of the description of each method - which can also be determined 
through the reflection interface - is the command needed to invoke this method. Because 
script 512 can determine the code needed to invoke any method and, as described above, 
can generate data values suitable to provide as inputs to that method, script 512 can 
generate code to call any method in the bean. 

In the preferred embodiment; directed at load testing, the order in which the 
methods of a bean are called is not critical to an effective test. Thus, script 512 can 
automatically generate useful test code by invoking each method of the bean. The order 
in which the methods are invoked does not matter if the only parameter that is measured 
is the time it takes the methods to execute. 

More sophisticated tests can be automatically built by relying on the prescribed 
pattern for the language. In Sun JAVA, entity beans for controlling access to a database 
should have methods that have a prefix "set" or "get". These prefixes signal that the 
method is either to write data into a database or to read data from the database. The 
suffix of the method name indicates which value is to be written or read in the database. 
For example, a method named setSSN should perform the function of writing into a 
database a value for a parameter identified as SSN. A method named getSSN should read 
the value from the parameter named SSN. 

By taking advantage of these prescribed patterns, script 512 can generate code to 
exercise and verify operation of both methods. A piece of test code generated to test 
these methods would first exercise the method setSSN by providing it an argument 
created by data generator 518. Then, the method getSSN might be exercised. If the get 
method returns the same value as the argument that was supplied to the set method, then 
it can be ascertained that the database access executed as expected. 

For many types of enterprise wide applications, the beans most likely to be 
sensitive to load are those that access the database. Thus, testing only set and get 
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methods provides very useful load test information. 

However, the amount of testing done can be expanded where required. Some 
beans also contain methods that create or find rows in a database. By convention, 
methods that create or find rows in a database are named starting with "create" or "find." 
Thus, by reflecting the interface of the bean, script 512 can also determine how to test 
these methods. These methods can be exercised similarly to the set and get methods. 
The properties revealed through the application interface will described the format of 
each row in the database. Thus, when a create method is used, data can be automatically 
generated to fill that row, thereby fully exercising the create method. 

In a preferred embodiment find methods are exercised using data from a user 
su pplied data table 520. Often, databases have test rows inserted in them specifically for 
testing. Such a test row would likely be written into data table 520. However, it would 
also be possible to create a row, fill it with data and then exercise a find method to locate 
that row. 

Once the commands that exercise the methods of an EJB technology based 
software component are created, script 512 can also insert into the client test code 516 the 
commands that are necessary to record the outputs of the test. If a test is checking for 
numbers of errors, then test code 516 needs to contain instructions that record errors in 
log 216. Likewise, if a test is measuring response time, the- test code 516 must contain 
instructions that write into log 216 the information from which response time can be 
determined. In the described embodiment, all major database functions can be exercised 
with no user supplied test code. In some instances, it might be possible to exercise all 
the functions with ail test data automatically generated. All the required information 
could be generated from just the object code of the application under test. An important 
feature of the preferred embodiment is that it is "minimally invasive" - meaning that very 
little is required of the user in order to conduct a test and the test does not impact the 
customer's environment. There is no invasive test harness. The client code runs exactly 
like the code a user would write. 
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In some scenarios, it will be necessary or desirable for a user to insert specific 
steps into the test code 516, Test system 1 10 allows for the user to edit test code 516. 
For example, some beans will perform calculations or perform functions other than 
database access, hi these instances, the user might chose to insert test code that exercises 
these functions. However, because bottlenecks in the operation of many N-tiered 
applications occur in the entity beans that manage database transactions, the simple 
techniques described herein are very useful. 

The results of any tests are stored in log 216. FIG. 2 shows log 216 as a separate 
hardware element attached to network 122. Log 216 could be any type of storage 
traditionally found in a network, such as a tape or disk drive attached to a computer 
server. For simplicity, it is shown as a separate unit, but could easily be part of any other 
server in the network. 

Many types of data could be stored in log 216. For example, there are several 
possible ways that "response time" could be measured. One way is that the total time to 
execute all the methods in a bean could be measured. Another way is that the start up 
time of a bean could be measured. The startup time is the time it takes from when the 
bean is first accessed until the first method is able to execute. Another way to measure 
response time is to measure the time it takes for each method to execute. As another 
variation, response time could be measured based on how long it takes just the "get-" 
methods to execute or just the "set-" methods to execute. 

Different measurements must be recorded, depending on which measure of 
response time is used. For example, if only the total response time is required, it is 
sufficient if the test code simply records the time that the portion of the test code that 
exercises all the methods starts and stops executing. If the startup response time is 
required, then the client test code must record the time that it first accesses the bean under 
test and the time when the first method in the test sequence is ready to be called. On the 
other hand, if the response time is going to be computed for each method, the client test 



code must record the time before and after it calls each method and some indication of 
the method being called must also be logged. Similar information must be recorded if 
responses of just "get-" or "set-" functions are to be measured, though the information 
needs to be recorded for only a subset of the methods in these cases. 

In addition, when there are multiple users being simulated, there are multiple 
values for each data point. For example, if test system 110 is simulating 100 users, the 
time that it takes for the bean to respond to each simulated user could be different, 
leading to up to 100 different measurements of response time. The response time for 1 00 
users could be presented as the maximum response time, i.e. the time it takes for all 100 
simulated users to finish exercising the bean under test. Alternative, the average time to 
complete could be reported as the response time. As another variation, the range of 
values could be reported. 

In the preferred embodiment, the client test code 516 contains the instructions that 
record all of the information that would be needed for any possible measure of response 
time and every possible display format. The time is recorded before and after the 
execution of every method. Also, an indication that allows the method to be identified is 
recorded in log 216. To support analysis based on factors other than delay, the actual and 
expected results of the execution of each method are recorded so that errors can be 
detected. Also, the occurrences of exceptions are also recorded in the log. Then, data 
analyzer 218 can review the log and display the response time according to any format 
and using any definition of response time desired. Or the data analyzer can count the 
number of exceptions or errors. 

Once the data is stored, the user can specify the desired format in which the data 
is to be presented. Data analyzer 218 accepts commands from the tester concerning the 
format of the output and analyzes the data appropriately. In a preferred embodiment, test 
system 110 has the ability to present the results of tests graphically to aid the tester in 
understanding the operations - particularly performance bottleneck - of application under 
test 114. 
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Data analyzer 218 generates output in several useful formats. One important 
output is a response time versus load graph. Log file 216 contains the starting and 
stopping times of execution tests for a particular test case. The test case includes the 
same measurements at several different load conditions (i.e. with the test engines 
214A...214C simulating different numbers of simultaneous users). Thus, data analyzer 
can read through the data in log 216 and identify results obtained at different load 
conditions. This data can be graphed. 

Another useful analysis is the number of errors per second that are generated as a 
function of the number of simultaneous users. To perform this analysis, test code 516 
could contain instructions that write an error message into log 216 whenever a test 
statement produces an incorrect result. In the database context, incorrect results could be 
identified when the "get" function does not return the same value as was passed as an 
argument to the "set" function. Or, errors might be identified when a bean, when 
accessed, does not respond or responds with an exception condition. As above, data 
analyzer 218 can pass through the log file, reading the numbers of errors at different 
simulated load conditions. If desired, the errors can be expressed as an error count, or as 
an error rate by dividing the error count by the time it took for the test to run. 

Some examples of the types of outputs that might be provided are graphs 
showing: transactions per second versus number of users; response time versus number of 
users; exceptions versus numbers of users; errors versus numbers of users; response time 
by method; response time versus run time and transactions per second versus run time. 
Different ways to measure response time were discussed above. In the preferred 
embodiment, a transaction is defined as the execution of one method, though other 
definitions are possible. 

Run time is defined as the total elapsed time in running the test case, and would 
include the time to set up the execution of EJB technology based software components. 
Viewing response time as a function of elapsed time is useful, for example, in revealing 
problems such as "memory leaks". A memory leak refers to a condition in which 


portions of the memory on the server running the application under test Rets used for 
unproductive things. As more memory is used unproductively. there is less memory 
available for running the application under test and execution slows over time. 
Alternatively, viewing results in this format might reveal that the application under test is 
effectively utilizing caching. If caching is used effectively, the execution time might 
decrease as elapsed time increases. Turning now to FIG. 6, operation of test system 1 10 
from the tester's perspective is illustrated, FIG.6 illustrates the tester interface to be a 
traditional web browser interface as it would appear on the terminal of PC 152 or 154. 
This type of interface is the result of using a web server to implement GUI 150 with 
commercially available browser software on the tester PC's 152 and 154. 

Element 612 is the browser toolbar. The browser provides the ability for the 
tester to perform such functions as printing and connecting to pages presented by various 
servers in the system. The tester performs these functions through the use of the tool bar 
612. Field 614 indicates the server to which the PC 152 or 154 is currently connected. In 
the illustrated example, field 614 contains the address on network 122 for the server 
containing GUI 150. 

Window 610 is the area of the screen of PC 152 or 154 where information 
specific to test system IIP is displayed. As with traditional web based applications, this 
information is displayed as the result of HTML files that are downloaded over network 
122. Certain elements displayed in window 610 represent hyperlinks, which when 
accessed by a tester cause other pages to be downloaded so that different information is 
displayed for the tester. Other elements displayed in window 610 represent data entry 
fields. When a human tester fills in these fields, information is submitted to test system 
110. 


Tabs 616 represent choices a tester can select for operating mode. FIG. 6 shows 
three tabs. There is one tab for "setup", one for "test case" and one for "results". Each of 
the tabs is hvperlinked. These hyperlinks comiect to pages on servers in the network that 
are programmed to manage the test system through a particular phase. Setup is for 
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configuring the test system, such as by identifying addresses for servers on network 122 
that contain test engines. 'Test case" is for creating and running a particular test case, 
which in the preferred embodiment means test code for a particular bean and parameters 
specifying how that test code is to run. "Results" is for viewing the results of a test case 
that has executed. In FIG. 6. the "SETUP" tab is shown selected, meaning that the 
balance of window 610 displays hyperlinks and fields that are appropriate for entering 
data or commands for the SETUP phase. 

Region 618 contains a set of hyperlinks. These hyperlinks present a list of 
choices to the user about what can be setup. Selecting one of these hyperlinks will 
change the information appearing in region 620. It is well known in the art of web based 
software to have hyperlinks on one part of window change information appearing in 
another part of the window. 

In the Example of FIG. 6, the hyperlink "Machine" in region 618 has been 
selected. Therefore, region 620 contains fields that can be used to identify a machine to 
be added to test system 110. In FIG. 6, a screen to add a client host computer is shown. 
A client host computer acts as a test engine 214A. ..214C. 

Region 618 also contains hyperlinks for "Projects" and "Data Tables". As 
described above, test system 110 can be used by multiple testers or can be used to test 
multiple applications under test. To segregate the information relating to various tasks, a 
"Project" is defined. All information relating to a particular project is identified with the 
project. In this way, test system 110 can identify information that is logically related. 
For example, test code developed to test a particular bean in an application will be given 
the same project name as test code to test other beans in that application. In this way, 
general information about the application - such as the path to the server through which 
the application can be accessed is stored once with the project rather than with each piece 
of test code. As another example, the test results can be associated with the test code that 
generated them through the use of projects. 
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The hyperlink for "Data Tables" allows the tester to identify to the system where 
data tables are stored to test specific beans. In general, the tester will create the data 
tables by hand or automatically apart from test system 110 based on the features that the 
tester desires to have tested. The data tables will be stored in files on a computer on 
network 122. Before use by test system 1 10, the tester must provide the network address 
for that file. 

Field 622 is a menu field. It is a drop down menu box. When accessed, it will 
display a menu of choices of the actions the tester can specify for test system 110. The 
contents of the menu is context sensitive, meaning that for any given page, only the menu 
choices that are appropriate are displayed. Actions that a user might want to choose from 
the menu are things such as edit, delete, add new, rename, etc. 

Turning now to FIG. 7, a screen of tester PC 152 or 1 54 is shown when the TEST 
CASE tab selected. Selecting the TEST CASE tab allows the tester to specify what is to 
be tested and how the test is to be conducted. In the example of FIG. 7, window 610 
contains information that describes a test case. This particular page is displayed when the 
"edit" has been selected in menu field 622. Field 710 indicates the name of the project 
to which the test case is associated. Field 710 is a screen display element known as a 
drop down box. When activated by a tester, field 710 will become a list of all projects 
that have been previously created by and tester using test system 110. As shown in FIG. 
7, a project called "Demo Project'' is being used. 

Field 712 identifies the particular test case by name. Field 712 is also a drop 
down box, allowing previously defined test cases to be selected from a list. In addition, 
one of the options that appears when the drop down box is accessed is "<New Test 
Case>". When selected, a new test case is created and information about this test case 
can be entered. This type of menu is common for programs with graphical user interfaces 
and is used at several points in the interface for presenting choices to a human tester. 


In the example of FIG. 1, a test case "Customer Test" has been created and 


information has already been entered. In region 714, there are a series of fields in which 
data can be entered or changed to define or modify the test case. In field 716, a name can 
be provided for the test case. This name allows the test case to be referenced in other 
parts of the test system. 

In field 718, a description of the test case can be entered. This description is not 
used by the automatic features of the test system, but can act as a note to a human tester 
to signify what the test case does or why it was created. Field 720 is likewise not used by 
the automated system. However, this field holds the name of an individual who authored 
the test case to facilitate other administrative functions. 

Field 722 is a "radio button" type field. It presents a tester with a list of choices, 
only one of which can be selected at a time. In this example, field 722 allows the tester 
to specify the type of test. As previously described, code generators 212A and 212B 
contain a plurality of scripts 512 that generate test code. As described above, the script 
assembles templates and generate command lines for a particular type of test that is to be 
conducted. Thus, the tester must specify the test type in order to allow the appropriate 
script to be selected. In this example, only two types of tests are presented as options to a 
tester - a load test and a functional test. These types of tests were discussed above. Field 
724 allows the tester to specify a "deployment descriptor." In the JAVA language, every 
bean has associated with it a deployment descriptor. The deployment descriptor is a file 
that identifies the bean and defines the parameters with which the EJB technology based 
software component will be deployed on the applications server. Examples of the 
parameters are the number of instantiations of the EJB technology based software 
component with which to start the applications server (sometimes called a "pooling 
number") and how much memory is allocated to the bean. These functions are performed 
by the container 130. 

The tester provides the deployment descriptor by entering the path to a file on 
network 122. The test system 1 10 reads the deployment descriptor to find the name of 
the bean under test and then to access the bean through reflection to determine its 
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methods and properties. 


Field 726 allows the tester to specify the type of data to be used in creating the 
test code 516. As described above, in the preferred embodiment, the data can be 
automatically generated by test system 1 10 or can be supplied through a data table. For 
automatic data generation, the data can be generated by using the maximum possible 
value of each variable, the minimum possible value or a value randomly selected between 
the maximum and the minimum. Alternatively, the data can be specified by a data table. 
In FIG. 7, field 726 indicates that tester desires test system 1 1 0 to generate data using the 
data table named dd.csv. If the tester had wanted the test system to automatically 
generate data, the tester would specify in field 726 whether the data was to be generated 
randomly or whether the maximum or minimum values were to be used. 

Field 728 allows the user to specify the server on which the application under test 
runs. FIG. 1 shows that the beans 132 of the application under test are within a container 
130. In this context, "server" refers to the software that creates the container for the 
application. For each platform independent language, there are a limited number of 
commercially available servers. Test system 1 10 contains script files that will generate 
the appropriate test code for any server. While most of the client test code will be server 
independent, it is possible that servers will implement certain functions in unique ways. 
Thus, there needs to be script files that account for the functions that are implemented 
differently on certain servers. 

FIG. 8 shows a screen display when the tester has used menu field 622 and 
selected to have the deployment descriptor for a test case to be displayed. If desired, the 
tester can then edit the deployment descriptor to try alternative configurations. 

FIG. 9 shows a screen display when a tester has selected from menu field 622 to 
have the generated test code displayed. In FIG. 9, the test code 516 is identified as 
"client code" because it will simulate the operation of a client 120 of the application 
under test 1 14. The displayed code corresponds to the code generated for the project and 
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test case identified in fields 710 and 712, In this screen, the tester can also edit the test 
code. 

One instance when a tester might desire to edit test code is when most of the 
commands in the test code can be automatically generated, but certain commands must 
have specific data values for the application under test to function properly. The tester 
could have test system 110 automatically generate the test code, but then manually edit 
the few data values that matter. As an example, a particular bean might include a method 
that processes positive and negative values differently. The tester might know that 
processing negative numbers takes longer and therefore change a randomly generated 
value to a negative number. 

An alternative scenario in which a tester might wish to edit test code 516 is when 
the bean being tested contains methods to be tested other than those that follow a pattern, 
such as the "set", "get", "create" and "find" methods described above. The tester might 
create test code that tests methods that do not follow the pattern. This code could then be 
inserted by the human tester into the generated test code. 

FIG. 10 shows a screen display for another function performed by a human tester 
while specifying the TEST CASE, also selected through menu field 622. FIG. 10 is a 
screen display used when the test case is to be run. The project and specific test case is 
specified by entries in fields 710 and 712. Information about how to run the test case is 
entered in region 1010. 

Region 1010 contains several fields. Field 1012 indicates the name of the file in 
log 216 where the results of executing the test case will be stored. In this way, data 
analyzer 218 will be able to locate the data for a particular test case to analyze and 
present to a tester in the desired form. 

Field 1014 gives the URL - or network address - for the application under test. 
This information could be identified by iising the name for a particular machine that was 
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previously set-up by the human tester. Field 1016 Rives the URL - or network address - 
for a server to use as a test engine. Again, the server could be identified by using the 
name for a particular machine that was previously set-up by the human tester. The screen 
displayed in FIG. 10 is used for an embodiment of the invention where all test 
simultaneously executing copies of the client test code are run on a single machine. If 
test system 110 includes multiple test engines and automatically schedules execution of 
test code as described in conjunction with FIG. 4 above, then field 1016 is not required. 

Field 1018 gives the maximum number of concurrent users to be simulated at one 
time during the execution of the test case. Field 1020 allows the user to specify the rate at 
which the number of simultaneous users will be simulated during a test. In the example of 
FIG. 10, the test case will be completed after 100 users have been simulated 
simultaneously and the number of simultaneous users will increase by 10 each time the 
test code is run. For the examples given here. 10 copies of the client test code shown in 
FIG. 9 will first be executed simultaneously. Then 20 copies of that test code will be 
executed simultaneously. The test code will be repeated for 30. 40, 50, 60, 70, 80, 90 and 
100 simultaneous users before the test case is completed. 

After the test case has been run, the tester can view and analyze the results. The 
human tester can access the functions of the test system 1 1 0 that display and analyze 
results by selecting the RESULTS tab from tabs 616. FIG. 11 shows a page displayed 
when the RESULTS tab is selected. The page shown in FIG. 1 1 is for when the tester has 
requested to see summary data through use of menu field 622. 

Fields 710 and 712 are also present on the page shown in FIG. 11. This 
information is used by the test system to locate the appropriate data to display. In 
addition, there is a field 1110 that allows the user to enter the name of the results file to 
display. As described in conjunction with FIG. 10, the user specifies in field 1012 the 
name of a results file to hold the results of a particular run of a test case. The name of the 
results file for desired run of the test is entered in filed 1110. 
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FIG. 1 1 shows that window 610 contains a region 1112 that lists the results of a 
run of a particular test case in summary fashion. Part of the information in region 1 112 is 
simply a display of information input by the tester when editing or running the test case. 
For example, the target container, or the container 130 of application under test 114 is 
listed. The maximum number of concurrent users simulated during the test is also 
displayed. The file containing the test data use for the run of the test case that generated 
the results is also displayed as is the deployment descriptor. These values are displayed 
for ease of use by the human tester. 

Region 1 112 also includes information that was developed by data analyzer 218 
from the data gathered for the specified run of the test case. In FIG. 11, the pieces of 
information that are displayed are the average response time and the maximum response 
time. As described above, as a test case runs, the start and stop times of the execution of 
the test code is recorded in log 216. When the test code is run multiple times, each time 
simulating a different number of users, the start and stop time is recorded for each 
number of users. Data analyzer can determine the response time by simply computing 
the difference between the start and stop times. Once values are determined, they can be 
averaged or the maximum can be identified. 

FIG. 12 shows a different page that can be selected by the user to see the results in 
graphical format. As above, fields 710. 712 and 1110 allow the user to specify which run 
of a particular test case is used to create the results. The graphical display in FIG. 12 is a 
graph showing numbers of transactions per second as the dependent variable with the 
number of simultaneous users as the independent variable. As described above, the 
information needed to compute the data values for this graph is stored in log 216 after the 
test case is run and data analyzer 218 can retrieve it. For creation of the graph in FIG. 12, 
transactions per second was defined as the average number of methods executed per 
second per user. This value is essentially the reciprocal of response ti me. 

FIG. 13 shows a screen useful for displaying results to a human tester in a slightly 
different embodiment. As with FIG. 12, the screen display in FIG. 12 is accessed when 
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the "RESULTS" tab is selected. Also like in FIG. 12. the page shown in FIG, 13 
includes fields 710. 712 and 1 1 10 that allow the human tester to specify which results are 
to be displayed. 

The page shown in FIG. 13 includes an alternative way for the user to specify the 
format of the display. The screen in FIG. 13 includes menu fields 1310 and 1312. Menu 
field 1310 allows the tester to specify the manner in which response time is to be 
calculated. In FIG. 13, a value of "total" has been selected in field 1310. As described 
above, the "total" response time is measured as the time from first access of a bean until 
all methods of the bean have been exercised. Other choices in menu field 1310 allow a 
' tester to specify 7 that result be displayed for different measures of response time. As 
described above, the presently preferred embodiment can measure response time just on 
the start up time or response time for individual methods or for get- functions and set- 
functions. 

Field 1312 allows a user to specify the format of the display. In FIG. 13, the 
display is in HiLo format. In this format results are displayed as bars, such as bar 1316. 
Each bar spans from the fastest response time to the slowest response time. A tick mark 
showing the average is also included in the illustration of FIG. 13. Other choices in menu 
field 1312 would, for example, allow the human tester to see results in a line chart format 
as in FIG. 12 or in tabular format. 

Turning to FIG. 14, results in a tabular format are shown. Field 1312 indicates 
that the display format of "Log File" has been selected. This format corresponds to the 
list shown in region 1412. The list contains a column for the name of the methods in the 
bean being tested. In this example, the data shown for each method reveals the 
minimum, maximum and average execution time for that method. 

As described above, test system 110 measures response time at various load 
conditions. The displayed data represents response times at a particular load condition. 
Thus, to make the list, the tester must specify the load condition for which data is to be 
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displayed. To allow this selection to be made, the page displayed in FIG, 14 contains a 
field 1410. In this field, a user can enter the load condition for which data is to be 
displayed. In this example, the human tester has entered a value of 500, indicating that 
500 threads executing the test code were initiated in order to obtain the displayed data. 

Having described the structure of test system 110 and giving examples of its 
application, several important features of the test system 1 10 can be seen. One feature is 
that information about the performance of an application under test can be easily 
obtained, with much of the data being derived in an automated fashion. A software 
developer could use the test system to find particular beans that are likely to be 
performance bottlenecks in an application. The developer could then rewrite these beans 
or change their deployment descriptors. For example, one aspect of the deployment 
descriptor indicates the number of copies of the bean that are to be instantiated within 
application under test 1 14. The developer could increase the number of instantiations of 
a bean if that bean is the bottleneck. 

The test system described herein provides an easy and accurate tool to test EJB 

technology based software components for scalability. It creates a user specified number 
of virtual iisers that call the EJB technology based software component while it is 
deployed on the applications server. The tool does this by inspecting the EJB technology 
based software component under test and automatically generating a client test program, 
using either rules based data or data supplied by an a human tester, and then 
multithreading the client test program to drive the EJB technology based software 
component under test. The result is a series of graphs reporting on the performance 
versus the number of users, which provide useful information in an easy to use format. 

Another feature of the invention is that the tests are run without requiring changes 
in the application under test or even the installation of special test agents on the server 
containing the software under test. The generated test code 516 exercises the bean in the 
a pplication under test using remote procedure calls. 


- 126 - 


Another feature of the described embodiment of test system 110 is that it is 
scalable. To increase the number of tests that could simultaneously be run or the size of 
the tests that could be run, more test engines could be added. Likewise, more code 
generators could be added to support the simulation of a larger number of simultaneous 
users. The specific number of copies of each component is not important to the 
invention. The actual number of each component in any given embodiment is likely to 
vary from installation to installation. The more users an application is intended to 
support, the more test engines are likely to be required. 

Another feature of the described embodiment is that testing is done on the 
simplest construct in the application under test - the beans in the illustrated example. 
There are two benefits to this approach. First, it allows tests to be generated very simply, 
with minimal human intervention. Second, it allows a software developer to focus in on 
the point of the software that needs to be changed or adjusted in order to improve 
performance. 

It should be appreciated that displays of specific kinds of information have been 
described. Various other analyses might be performed. It was described that response 
times and error rates as a function of load could be graphed for display to a human tester 
for further analysis. One enhancement that might be made to test system 110 is that the 
data analyzer 218 could be programmed to perform further analysis. It has been 
recognized that, as the load increases, there is often some point at which the performance 
of the system drastically changes. In some instances, the time to complete a transaction 
drastically increases. A drastic increase in transaction processing time indicates that the 
system was not able to effectively handle the load. However, a decrease in processing 
time can also indicate the load limit was reached. Sometimes, a system under test will 
respond with an error message more quickly than it would take to generate a correct 
response. Thus, if the only parameter being tracked is response time, a decrease in 
processing time as a function of load can also signal that the maximum load has been 
exceeded. Of course, an increase in errors or error rate can also signal that the maximum 
load was exceeded. Data analyzer 218 could be used to identify automatically a 
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maximum load for a particular test case. By running multiple test cases, each test case 
focusing on a different bean, test system 1 10 could automatically determine the bean that 
is the performance bottleneck and could also assign a load rating to application under test 
114. 


Having described one embodiment, numerous alternative embodiments or 

variations might be made. For example, it was described that test system 110 
automatically generates test code to exercise beans that follow a pattern for database 
access. These beans are sometimes called "entity beans." In general, there will be other 
beans in an application that perform computations on the data or that control the timing 
of the execution of the entity beans. These beans are sometimes called "session beans." 
Session beans are less likely to follow prescribed programming patterns that make the 
generation of test code for entity beans simple. As a result, the automatically generated 
test code for session beans might not fully test those beans. In the described 
embodiment, it is expected that the human tester supply test code to test session beans 
where the automatically generated tests are inadequate. 

One possible modification to the described embodiment is that the completeness 
of tests for session beans might be increased. One way to increase the accuracy of tests 
for session beans would be to capture data about the execution of those beans during 
actual operation of the application under test 1 14. This data could allow an automated 
system to determine tilings like appropriate data values, which might then be used to 
build a data table. Or, the captured data could allow the automated system to determine 
the order in which a session bean accesses other session beans or entity beans to create a 
realistic test. 

Also, as described, test code is generated to test a particular bean, which is a 

simple construct or "component" of the application under test. The testing could focus 
on different constructs, such as specific methods in a bean. Test code could be generated 
to test specific methods within beans. Or, it was described that the system records start 
and stop time of the execution of the test code. The times of other events could be 
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recorded instead or in addition. For example, start and stop times of individual methods 
might be recorded, allowing performance of individual methods to be determined. 

Alternatively., the complexity of the constructs being tested could be increased. 

Multiple beans might be tested simultaneously to determine interactions between beans. 
For example, multiple test cases might be executed at the same time, with one test case 
exercising a specified instances of one bean and a different test case exercising a 
specified number of instances of a second bean. 

As another example of a possible variation, it was described that a human tester 

can insert code into a template to do such things as put the application under test into a 
predictable state. Lines of code might be inserted directly, for example by the user 
simply typing the lines of code. Or, the tester might insert a "tag" into the template. The 
tag would identify a code segment stored elsewhere within the test system. In this way, 
the same code segment could be included at multiple places in the template or in multiple 
templates. 

As another example of possible variations, the number of templates used to 

construct test code might be varied. One possibility is that each template contains all of 
the steps needed to initialize, run and terminate a test. Thus, test code would be created 
by filling in a single template. Alternatively, each template might contain only the steps 
needed to perform one function, such as initialization, testing or termination. In this 
implementation, test code would be created by stringing together multiple templates. 

Also, it was described that in running a test that a number of simultaneous users is 
"synchronized". Simultaneous users are simulated by synchronizing copies of the test 
code on different servers and on the same server. The term "synchronized" should not be 
interpreted in so limited a way as to imply that multiple copies are each performing 
exactly the same function at exactly the same time. Thus, when described herein that 
execution is synchronized, all that is required is that each copy of the code is making 
requests of the application under test during the window of time when the test is being 
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executed. Some copies of the code will likely start execution sooner or end sooner than 
the others. However, as long as there is overlap in the timing of execution, the test 
programs can be said to be synchronized or running concurrently. 

As a further variation, it was described that the test system 110 provides outputs 

indicating the performance of an application under test as a function of load. These 
outputs in graphical or tabular form can be used by an application developer to identify a 
number of concurrent users at which problems with the application are likely to be 
encountered. Potential problems are manifested in various ways, such as by a sudden 
change in response time or error rate as a function of load. Test system 1 1 0 could readily 
be programmed to automatically identify patterns in the output indicating these problem 
points. 

Another useful modification would allow test system 110 to aid in identifying 

settings for various parameters in the deployment descriptor. As described above, the 
deployment descriptor for a bean identifies parameters such as memory usage and a 
"pooling number" indicating the number of instances of a bean that are created at the 
initialization of an application. These and other settings in the deployment descriptor 
might have an impact on the performance time and maximum load that an application 
could handle. One use of the test system described above is that it allows a test case to be 
repeated for different settings in the deployment descriptor. A human tester can analyze 
changes in performance for different settings in the deployment descriptor. However, 
test system 1 10 could be programmed to automatically edit the deployment descriptor of 
a bean by changing parameters affecting pooling or memory usage. Test system 110 
could then automatically gather and present data showing the impact of a deployment 
descriptor on performance of an application. 

Even higher levels of automation could be achieved by test system 110. For 

example, test system 110 might test the beans in an application and analyze the results of 
testing each bean. Test system 110 might identify the bean or beans that reflect 
performance bottlenecks (i.e. that exhibited unacceptable response times for the lowest 
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numbers of simultaneous users). Then, test system 1 10 could run tests on those beans to 
find settings in the deployment descriptors that would balance the performance of the 
beans in the application (i.e. to adaptively adjust the settings in the deployment 
descriptors so that the bottleneck beans performed no worse than other beans.) 

It should also be appreciated that computer technology is rapidly evolving and 

improved or enhanced versions of the hardware and software components making up the 
application under test and the test system are likely to become available. It should also 
be appreciated that the description of one device in a class is intended to be illustrative 
rather than limiting and that other devices within the same class might be substituted with 
ordinary skill in the art. For example, the application under test was described in the 
context of a conventional application accessed through a client on a PC using a web 
browser as a graphical user interface. It should be appreciated, though, that if the clients 
are intended to be household appliances with micro-controllers, a different interface 
might be readily substituted for the graphical user interface. 

Also, it was described that FIG. 1 1 shows summary data of a test after execution 

is completed. It will be appreciated, though, that data on a test case execution might be 
displayed to a human tester while a test is in process. For example, the summary screen 
might contain a field that shows the percentage of the test case that is completed. This 
value would update as the test runs. Likewise, the values for average and maximum 
response time could be updated as the data is gathered. 

Also, it was described that the objects being tested are EJB technology based 

software components, which are written in the Java language. The same techniques are 
equally applicable to applications having components implemented in other languages. 
For example, applications written according to the COM standard might be written in 
Visual Basic and applications written for the CORBA standard might be written in C++. 

Regardless of the specific language used, these standards are intended to allow 
separately developed components to operate together. Thus, each must provide a 
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mechanism for other applications, such as test system 1 10, to determine how to access the 
methods and properties of their components. However, there could be differences in the 
specific commands used to access components. 

In one embodiment, code generator 212 is implemented in a way that will make it 
easy to modify for generating test code for applications written in a different language. 
Specifically, code generator 212 stores intermediate results as a symbol table that is 
independent of the specific language used to program the application under test. The 
symbol table lists methods and properties for the component being tested. When to 
access these methods and what data to use for a particular test and what kinds of data to 
record can be determined from the information in the symbol table and input from the 
user. Thiis, much of the functioning of code generator 212 is independent of the specific 
language used to implement the application under test. 

In this way, the language specific aspects of code generator 212 are easily 
segregated and represent a relatively small part of the code generator 212. In particular, 
language specific information is needed to access the application under test to derive the 
information for the symbol table. Language specific information is also needed to format 
the generated client test code. But, it is intended that these parts of code generator 212 
could be replaced to allow test system 1 10 to test applications written in other languages. 
Also, it is possible that test system 110 will contain multiple versions of the language 
specific parts and the user could specify as an input the language of the application under 
test. 

Therefore, the invention should be limited only by the spirit and scope of the 

appended claims. 
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